Python Flask Date Update Real-time

I am building a web app with Python Flask with JavaScript. I am a beginner of Javascript. The process I do now: In Flask Python code, 1. I get data by scrapping the web (numeric da

Solution 1:

This is a classic "push" update problem. Your web server is updated with information that you would like to frequently update (or "push") to your web client front-end.

As PJ Santoro suggests, you can repeatedly poll your server (every 10 seconds say) along the lines of "Is there new data? No. Okay. Is there new data? No. Okay. Is there new data? Yes, here it is! Great! Is there new data? ...." It is rather inefficient, but for low data volumes you might never see that problem.

A more efficient update system would have the server send an update signal only when new data is ready. There are two major ways to do that:

1. Long polling, also known as reverse AJAX and Comet. Your web page opens an AJAX request on a server URL such as /update_data with a really long timeout (say 30 minutes or an hour). When the server has data, it sends it through the waiting AJAX connection. There are complications with this, including managing concurrency on the server side (not historically Python's strong suit) and the fact that even the maximum timeout (~1 hour last I used this technique) might expire without data, and some error handling is needed to manage that. If you wan to try long polling / reverse AJAX, here is an example using Flask.

2. WebSockets. WebSockets are a much more recent approach to interactive and "push" updates between servers and clients. They're very well-used; projects like the Jupyter Notebook depend on them extensively. They're extremely efficient and well-fit for this kind of incremental update of client state, and the code is often much less Byzantine and confusing than retrofits like reverse AJAX. But...there are complexities with WebSockets too. Managing concurrency in the Flask server side is still a significant issue, for example. If you'd like to consider WebSockets as your update mechanism, here is an example of how to use them with Flask.

Either approach you use, if your data communicated over time grows, you will also need to structure the data transmitted on each update so that it's an incremental update. No mater how good the plumbing, if you transmit thousands of data points every update, it's not going to be fast. But, long polling and WebSockets plumbing should at least get you a great distance toward a legitimate real-time update capability.

Solution 2:

Well... in an ideal world you would separate this into 2 applications:

• Scrape & Parse Data From the Web (doesn't need to be exposed to the web)

• Deliver data to a user via a web app

You could use some sort of CI tool (e.g. Jenkins) to monitor and add the external data into a database and then your Flask app to serve this pre-processed data to your users.

If Steps 1-6 are relatively quick what you could do is setup an XHR endpoint within Flask and then use a setInterval() javascript function to call it on a interval to tell your application to update the data. E.g.:

setInterval(function() {
    var req = new XMLHttpRequest();
    req.open('GET', '/refresh_data?time=234', true);
    req.onreadystatechange = function(e) {
        if(req.readyState !== 4) {
            return;
        }
        if ([200, 304].indexOf(req.status) === -1) {
            console.warn('Error! XHR failed.');
        }
        else {
            data = JSON.parse(e.target.responseText);
            processData();
            updateChart();
        }
    };
    req.send();
}, 10000);  // time in milliseconds (e.g. every 10 seconds)

And have a flask endpoint like:

@app.route('/refresh_data')
def refresh_data():
    time = requests.args.get('time', type=int)
    if not time:
        return jsonify({status: 'error: incorrect parameters'})
    data = requests.get('http://api.getmoredata.com?time=%s' % time)
    # process the results...
    return jsonify({status: 'success', data: data})

Ideally you'd have some sort of a chart that has a refresh() method where you can pass it new data and just keep adding to it... (e.g. I'm a fan of D3 for stuff like this).

I'd probably remove this from you're init.py code, but it might be acceptable for the "first load". a flask app (e.g. any web app) responds to HTTP requests and nothing persists on the server between requests so a time.sleep() would not do much for you... if you want to run persistent code on your server you'd need to look into something like celery to manage background tasks.