Real-time Data Recovery using Multidirectional LSTM in Wireless Sensor Networks
By: Hermawan, Ade; Ginanjar, Rizki; Kim, Dong-Seong; Lee, Jae-Min
Dear Prof. Jae-Min Lee:
The review of the referenced manuscript, WCL2020-0737, is now complete. I regret to inform you that based on the enclosed reviews and my own reading of your manuscript, I am unable to recommend its publication in IEEE Wireless Communications Letters.
Your paper may not be resubmitted for review. We have received feedback on your manuscript from two experts. Both have expressed concerns on the novelty and the nature of technical evaluation to validate the performance of the proposed methods. Some important comments include
1. Lack of novelty since the framework of using LSTM ideas to compute missing data is well known. Reviewer 2 thinks that presenting the accuracy and computation time of these methods are rather shallow, instead, suggests to analyze the trade-off between accuracy and computation time in a rigorous manner, and also address problem statements that capitalize the existence of LSTM schemes to schedule infrequent updates of data from the sensors so as to increase energy-efficiency.
2. Claim on computational efficiency of DL methods is not well supported owing to the use high end GPUs, which may not go well with real time data recovery applications. Furthermore, Reviewer 1 thinks that complexity of the training process must also be discussed, and also should consider relevant baselines such as SOTA and other deep learning methods instead of using SARIMAX, which can not utilize future data for inferring past missing data points.
Based on the above comments, and my own reading of the manuscript, I am unable to recommend this manuscript for publication in IEEE Wireless Communication Letters.
The reviewers' comments are found at the end of this email.
Thank you for submitting your work to the IEEE Wireless Communications Letters. I hope the outcome of this specific submission will not discourage you from the submission of future manuscripts.
Dr. J Harshan
Editor, IEEE Wireless Communications Letters
Comments to the Author
In this paper, the authors presented an algorithm to recover missing data from sensors in real-time. Their main contribution is the use of biLSTM by incorporating future observations so as to recover the missing data points. While the approach is meaningful, I don't find it novel enough for IEEE WCL. Below are some of the major issues I found with this paper:
1) I find it hard to believe that DL in comparison to classical deterministic approaches is computationally efficient. The authors use Nvidia RTX 2070 which is a high end GPU and doesn't support the point that authors want to convey in terms of real time data recovery. However, they show superior performance in the evaluation section which again only includes SARIMAX. Authors should include comparison with SOTA and other deep learning/machine learning based approaches.
2) Also, since SARIMAX can not utilize future data for inferring past missing data points, in a way, it might be an unfair comparison. Authors should include a comparison with methods that utilize (or can utilize) future data points or, clearly mention that they are the first ones doing so which might not be the case anyway.
2) The paper doesn't clearly describe the training of LSTM/biLSTM architectures. Note that this will also add to the runtime of their method. One can argue that training has to be performed only once (which is probably what this paper is doing as mentioned in the evaluation section) but that is typically why Neural Networks are not very computationally efficient. I still expect the authors to compare the training time and then argue that we don¡¯t train the model anymore and only utilize it for inference.
3) The paper is not written clearly and there are minor typos. I believe that LSTM equations, since they are not proposed by the authors, can be taken out and the original paper can be cited here instead. Similarly biLSTM should also be cited since it is not the contribution in this paper.
In conclusion, authors should rethink whether their main contribution is better runtime or improved performance (as shown in the evaluation section). A more important problem would be where the sensor data doesn¡¯t come from a fixed distribution and the model has to be updated which is more practical from my point of view. However, in this case, training has to be done more carefully keeping in mind the increase in runtime.
Therefore, as of now, this paper is not a good fit for IEEE WCL in my opinion.
Comments to the Author
The paper uses an LSTM (Bi-LSTM) framework for filling the missing values in a time series data. The following are the comments:
1. The paper lacks novelty as it is well known that LSTM (and its variants) work well on time series data. The framework is well studied, and can be implemented pretty easily
(lot of online resources are available).
2. Although the performance in terms of accuracy is better in LSTM and Bi-LSTM compared to SARIMAX, they seem to have higher compute time. The authors do not comment on the trade-off that may exists between compute time and accuracy. The insights provided by the study is not clear.
3. An interesting problem would be to study when the sensors should send data. For example, if the next value can be easily predictable, the sensor need not burn energy to send the data.
4. In the experiments (related to weather), they seem to consider only three features. What is the rationale behind this?
5. There are several notations that are undefined. For example, \sigma_c and \sigma_h are undefined (they define \sigma which is never used). There are several errors in the paper. For example, "performance component analysis (PCA)" in line 18, page 2. There are several such errors. I request the authors to go through the paper
carefully once to fix all such errors and typos.