Serving New Layout With Time Expiring Cache

[kitsonswannff]

Hi There,

In my application I'm reading in data from an azure data store and then using it to make plots. And, I would like to refresh that data every 10 minutes. I've accomplished the refresh, but on initial page load it still shows the old data until the first update is triggered. With a 10 minute update users of the app think it's broken because they don't see the current data for 10 minutes.

I've looked at the examples here: https://plot.ly/dash/live-updates and here: https://plot.ly/dash/performance

I've implemented the layout as a function as suggested like this:

def get_layout():

    # function to get updated data
    data = get_data()

    # define the layout of the app
    layout = html.Div([
        
        # data is used in generating the layout here

        # I have an interval component in here driving the update

        # I am also storing the intermediate result as a json object in a hidden div as suggested in other posts
    ])

    return layout

app.layout = get_layout()

Even with the above function and the data being defined inside the function that serves the layout, instead of as a global variable, I still don't see the most recent data on initial load or page refresh. I've tried to implement a time expiring cache as suggested, but I can't seem to get it working.

I'm wondering if you could give a complete example that shows serving the layout from a function complete with a time-expiring-cache.

Also, side note the link here: https://plot.ly/dash/live-updates to time-expiring-cache is broken. Just an FYI.

Thanks!

[chriddyp]

I've tried to implement a time expiring cache as suggested, but I can't seem to get it working.

Could you paste an example of what you have tried so far?

[kitsonswannff]

Here's a minimum example of what I was trying:

# initialize the app
app = dash.Dash()
app.title = "Example Dash App"

cache = Cache(app.server, config={
    # try 'filesystem' if you don't want to setup redis
    'CACHE_TYPE': 'simple',
    'CACHE_DEFAULT_TIMEOUT': 60
})

app.config.supress_callback_exceptions = True

timeout = 60

# put everything in a function so it updates on page refresh
@cache.cached(timeout=timeout, key_prefix='current_layout'))
def get_layout():

    # get the data
    data = get_data()

    # define the layout of the app
    layout = html.Div([

        html.Div([html.H1('Example Dash App')], className="page-header"),

        # to refresh the data every five minutes
        dcc.Interval(
                id='interval-component',
                interval=10*60*1000
        ),

        # intermediate value to avoid re-querying the data in every callback
        html.Div(data.to_json(), id='intermediate-value', style={'display': 'none'}),
        
        # Plotting of graphs happens in here - removed to make example shorter


    ],className='container-fluid')

    return layout

app.layout = get_layout()

# update the intermediate value
@app.callback(dash.dependencies.Output('intermediate-value', 'children'),
    events=[dash.dependencies.Event('interval-component', 'interval')])
def update_json_data():

    # get updated data
    new_data = get_data()

    return new_data.to_json()

if __name__ == '__main__':
    app.run_server(port=8050)

This is the error I'm getting:

Traceback (most recent call last):
  File "/anaconda/lib/python3.6/site-packages/flask_cache/__init__.py", line 300, in decorated_function
    timeout=decorated_function.cache_timeout)
  File "/anaconda/lib/python3.6/site-packages/werkzeug/contrib/cache.py", line 317, in set
    pickle.HIGHEST_PROTOCOL))
_pickle.PicklingError: Can't pickle <class 'Div'>: attribute lookup Div on builtins failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app.py", line 262, in <module>
    app.layout = get_layout()
  File "/anaconda/lib/python3.6/site-packages/flask_cache/__init__.py", line 302, in decorated_function
    if current_app.debug:
  File "/anaconda/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/anaconda/lib/python3.6/site-packages/werkzeug/local.py", line 306, in _get_current_object
    return self.__local()
  File "/anaconda/lib/python3.6/site-packages/flask/globals.py", line 51, in _find_app
    raise RuntimeError(_app_ctx_err_msg)
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
to interface with the current application object in a way.  To solve
this set up an application context with app.app_context().  See the
documentation for more information.

[chriddyp]

Thanks @kitsonswannff !

Two issues going on here:
1 - dash_html_components and dash_core_components aren't pickle-able
2 - Flask caching is complaining about caching app.layout out of the app context.

To fix both of these issues, I recommend placing the computationally-expensive portions of app.layout in a separate function that can be cached. Here's a simple example:

import dash
from dash.dependencies import Input, Output
import dash_html_components as html
import dash_core_components as dcc
import datetime
import os
from flask_caching import Cache


app = dash.Dash(__name__)
cache = Cache(app.server, config={
    'CACHE_TYPE': 'filesystem',
    'CACHE_DIR': 'cache'
})
app.config.supress_callback_exceptions = True

timeout = 20

@cache.memoize(timeout=timeout)
def compute_expensive_data():
    return str(datetime.datetime.now())


def generate_layout():
    expensive_data = compute_expensive_data()
    return html.Div([
        html.H3('Last updated at: ' + expensive_data),
        html.Div(id='flask-cache-memoized-children'),
        dcc.RadioItems(
            id='flask-cache-memoized-dropdown',
            options=[
                {'label': 'Option {}'.format(i), 'value': 'Option {}'.format(i)}
                for i in range(1, 4)
            ],
            value='Option 1'
        ),
        html.Div('Results are cached for {} seconds'.format(timeout))
    ])

app.layout = generate_layout


@app.callback(
    Output('flask-cache-memoized-children', 'children'),
    [Input('flask-cache-memoized-dropdown', 'value')])
@cache.memoize(timeout=timeout)
def render(value):
    return 'Selected "{}" at "{}"'.format(
        value, datetime.datetime.now().strftime('%H:%M:%S')
    )


if __name__ == '__main__':
    app.run_server(debug=True)

[crazycodingcat]

Hi @kitsonswannff , I'm doing a similar dash application and have the exact same problem here. Just want to know if you had any success using flask-caching?

[crazycodingcat]

Just answering myself here:

Tried the approach @chriddyp suggested above and it is working fine now.

Took me some time to figure out that app.layout must be a function, i.e. app.layout = generate_layout, not app.layout = generate_layout(), otherwise dash seems to somehow cache the layout so it still shows the old data in every reload.

[kitsonswannff]

Hey, sorry I didn't get back to you here. That was exactly my problem and also took me forever to figure it out. Glad you sorted it. On Sep 25, 2017 8:51 PM, "crazycodingcat" <notifications@github.com> wrote: Just answering myself here: Tried the approach @chriddyp <https://github.com/chriddyp> suggested above and it is working fine now. Took me some time to figure out that app.layout must be a function, i.e. app.layout = generate_layout, not app.layout = generate_layout(), otherwise dash seems to somehow cache the layout so it still shows the old data in every reload. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#117 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ac1wAVAMXL9gVi_ZMYVOSGdRQ1A1f7Ujks5smHSmgaJpZM4O72uA> .

[howardski]

I tried using the separate function for caching as well as Chris pointed out, it worked, but if the computation is very heavy (takes a long time) the whole app seems to just spin (this happens when it takes longer to compute the data than the interval specified in the timeout argument). Is there a way we can have the cache timeout to wait until the computation is done ?

[0xAtomist]

Thanks for all the explanation Chris, however I am still a bit confused. I am currently doing something similar to OP where data is stored in a hidden div and updated at set intervals and also with a dropdown. Is this caching of functions supposed to completely erase the need for the hidden div and intervals and if so how do I keep my data up to date with a dropdown as well as the time-expiring cache, as there would be no dash object to output to?

[chriddyp]

Is this caching of functions supposed to completely erase the need for the hidden div and intervals and if so how do I keep my data up to date with a dropdown as well as the time-expiring cache, as there would be no dash object to output to?

I've tried my best to explain all of the different options with examples in this page in the user guide here: https://dash.plot.ly/sharing-data-between-callbacks. Could you check that out?

[ozagordi]

Hi @chriddyp.
I'm trying with the code in Example 3, and I have errors. First,

raise ValueError("cannot have a multithreaded and "
ValueError: cannot have a multithreaded and multi process server.

Then I changed the value of processes in app.run_server from 6 to 1 and I have a bunch of errors that end with

  File "/Users/ozagordi/miniconda3/envs/py36/lib/python3.6/site-packages/redis/connection.py", line 489, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 61 connecting to None:6379. Connection refused.

I'm running in a conda environment with python version : 3.6.1.final.0. The code you posted last Aug 17th (one year ago, wow!) works, though.

Thanks.
Osvaldo