[Solved] Forms take two attempts to submit, possibly related to writing files

[mabass]

Hi,

 

I'm running into an odd issue with two of my flask-based web-apps providing user-input forms.

 

EDIT: Ignore the "writing files" part in the title - more testing shows it's not related to this aspect.

 

The issue:

After submitting the form the page seems to accept the POST-request without hiccup, no error messages are displayed. However, none of the intended functionality actually takes place, the page containing the form simply reloads instead of properly redirecting. However, if attempting to submit a second time, everything works perfectly fine (*most of the time: sometimes it takes a third attempt).

 

After it worked once, forms will behave as expected. Reloading the page manually, i.e. resetting the form, causes the error to appear again.

 

This issue does not occur on localhost or on pythonanywhere.

 

Testing:

If you want to test it for yourselves it's done the easiest with https://mabass.tk/adx_abctrainer/
Enter anything in the first field and press the button next to it OR simply press the "Los!" button below.

 

Expected behavior:

Redirect to page showing images corresponding to the entered word, e.g. https://mabass.tk/adx_abctrainer/heliohost
 

Actual Result:

No response on first submit, page reloads.

The fact that errors are displayed for the second form below but not the first one is prove that the POST-request is registered.

The usage of two forms on one page isn't the issue here. This page is simply the easiest to reproduce the issue with.

 

For comparison, here's the identical app on pythonanywhere, working as intended:

http://wolfenswan.pythonanywhere.com/adx_abctrainer/

 

 

More testing:
I deployed a super simple flask app with two forms, one using wtfforms, the other pure html.

Irritatingly, these two seem to work just fine.

Edited November 4, 2020 by mabass

[Krydos]

If you look at the source of https://mabass.tk/flask.wsgi/adx_abctrainer/ you can see this on line 30

<form action="" method="post" novalidate class="form-inline">

when the action="" that means it will just post the data to the same page that you entered the data on. This might be fine depending on your code. If that same page's code examines the post data before outputting anything to the browser and then changes the headers to redirect to the second page it might work, but then the second page won't get the posted data. I guessing what you're intending to do is put the url to the second page such as action="/adx_abctrainer/heliohost"? I'm actually not sure why the pythonanywhere page works, because it has the same action="". Our servers are functioning the way html post forms are supposed to work.

[mabass]

Using action="" is generally the recommended way with Flask (and wtfforms), as the POST-request is then handled by the flask-routes, using redirect().

 

Also, the issue persists on pages with only one form.

 

The barebones examples works the exact same way as well, using action=""

@app.route('/form-1', methods=["GET", "POST"])
def index(i='None yet'):

    flash(f'1 Request:{request.method}')
    if request.method == "POST":
        flash(f'2 Input: {request.form.get("input")}')
        i = request.form.get("input")
        return redirect(url_for('.success',i=i))
    return render_template('form-vanilla.html',i=i)

@app.route('/form-2', methods=["GET", "POST"])
def form_2(i='None yet'):
    form = InputForm()
    flash(f'1 Request:{request.method}')
    if form.validate_on_submit():
        flash(f'2 Input: {form.word.data}')
        i = form.word.data
        return redirect(url_for('.success',i=i))
    return render_template('form-wtf.html',i=i,form=form)

@app.route('/success', methods=["GET"])
def success():
    i = request.args.get('i')
    return render_template('index.html',i=i)

I have a hunch that the issue might lie in the re-routing through flask.wsgi, as that's the only concernable difference between localhost, pythonanywhere und heliohost.

 

I'll try and see if changing action="" helps, but it might be tricky to realize.

Edited November 4, 2020 by mabass

[Krydos]

It might be the .htaccess rewrite that is messing with the routes then. I'm not sure. I'm not too familiar with flask. I figured out enough about it to write the basic example, and I know proxying flask through a dedicated web server like Apache is the recommended way to set up a production environment rather than running flask as the web server. Apache definitely does a much better job of handling ssl, etc.

[mabass]

That's def. true

 

I'm just stumped what could be causing this. Why would the script accept the POST without issues the first time but refuse the redirect, but then work almost all times afterwards - until the next forced reload of the page? And that only on heliohost, not on pythonanywhere(which also uses wsgi). So weird.

Edited November 4, 2020 by mabass

[Krydos]

I didn't realize pythonanywhere used wsgi too. I was assuming the python flask process was serving the http requests. Does python anywhere use proxy in the httpd.conf or use .htaccess rewrites? It might be that we need to set this up in httpd.conf instead of .htaccess. Something to try anyways if you want.

[mabass]

I'd assume the former, as I can't find any .htaccess files (though py-anywhere is more restrictive in what I can access, so they might be forcibly hidden).

 

And sure, I'd give that a shot.

 

Here's pyanywhere's full wsgi for completeness sake - not much different tbh.

# This file contains the WSGI configuration required to serve up your
# web application at http://<your-username>.pythonanywhere.com/
# It works by setting the variable 'application' to a WSGI handler of some
# description.
#
# The below has been auto-generated for your Flask project

import sys

# add your project directory to the sys.path
project_home = '/home/Wolfenswan/sandbox'
if project_home not in sys.path:
    sys.path = [project_home] + sys.path

# import flask app but need to call it "application" for WSGI to work
from app import app as application  # noqa

 

[mabass]

Did a bit more testing with networking tools and the problem more often than not corresponds with longish server delays.

The initial (failing) POST-request has nearly one full second waiting delay.

 

After running the app successfully once, the waiting's down to ~150-200ms.

 

Whenever, for whatever reason, the delay jumps above 500ms the app seems to break as described.

 

Not sure if there's much to be done about it besides switching to a VPS?

Edited November 4, 2020 by mabass

[Krydos]

I took a look at your /home/mabass/public_html/flask-live/.htaccess file and you had this

RewriteRule ^(.*)$ flask.wsgi/%{REQUEST_URI} [QSA,PT,L]

I changed it to this

RewriteRule ^(.*)$ flask.wsgi/$1 [QSA,PT,L]

and now it works 100% of the time for me.

[mabass]

Hm, I still get the issue occassionally; in any case I had switched %1 to %{Request... some time after I had first noticed the issue.

 

It's more pronounced on the other form - any "first" time the form is submitted, the delay hits ~1s; the second submit it works fine and the download link is displayed.

 

My best guess is that somehow my app causes a longer waiting-delay on heliohost than it does on e.g. pythonanywhere and flask's routing breaks as a result.

Maybe it's related to accessing (and in the case of the form linked above: writing) files?

 

That said, the issue seemed to be worse a few hours ago when Tommy's load was around ~10; now it's around ~5. That's anecdotal however.

Edited November 4, 2020 by mabass

[Krydos]

Oh, maybe. I changed the .htaccess and then tested it like 30 times in a row and it worked every single time on the first try so I thought that was the issue.

[Krydos]

Oh, this might be something

[Wed Nov 04 21:18:18.423483 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568] [2020-11-04 21:18:18,419] ERROR in app: Exception on /adx_abctrainer/Lebensmittel [GET]
[Wed Nov 04 21:18:18.423928 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568] Traceback (most recent call last):
[Wed Nov 04 21:18:18.423944 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
[Wed Nov 04 21:18:18.423956 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     response = self.full_dispatch_request()
[Wed Nov 04 21:18:18.423968 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
[Wed Nov 04 21:18:18.423979 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     rv = self.handle_user_exception(e)
[Wed Nov 04 21:18:18.423990 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
[Wed Nov 04 21:18:18.424002 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     reraise(exc_type, exc_value, tb)
[Wed Nov 04 21:18:18.424013 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
[Wed Nov 04 21:18:18.424024 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     raise value
[Wed Nov 04 21:18:18.424035 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
[Wed Nov 04 21:18:18.424047 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     rv = self.dispatch_request()
[Wed Nov 04 21:18:18.424058 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
[Wed Nov 04 21:18:18.424069 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     return self.view_functions[rule.endpoint](**req.view_args)
[Wed Nov 04 21:18:18.424080 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/home/mabass/public_html/flask-live/projects/adx_abctrainer/routes.py", line 42, in abctrainer_word
[Wed Nov 04 21:18:18.424092 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     return render_template('adx_abctrainer/word.html', word=word, form_2 = form_2, randomize=False, min=min_length, max=max_length, min_abs=WORD_LENGTH_MIN, max_abs = WORD_LENGTH_MAX)
[Wed Nov 04 21:18:18.424111 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/flask/templating.py", line 138, in render_template
[Wed Nov 04 21:18:18.432195 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     ctx.app.jinja_env.get_or_select_template(template_name_or_list),
[Wed Nov 04 21:18:18.432223 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/jinja2/environment.py", line 930, in get_or_select_template
[Wed Nov 04 21:18:18.432240 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     return self.get_template(template_name_or_list, parent, globals)
[Wed Nov 04 21:18:18.432271 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/jinja2/environment.py", line 883, in get_template
[Wed Nov 04 21:18:18.432285 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     return self._load_template(name, self.make_globals(globals))
[Wed Nov 04 21:18:18.432296 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/jinja2/environment.py", line 857, in _load_template
[Wed Nov 04 21:18:18.432307 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     template = self.loader.load(self, name, globals)
[Wed Nov 04 21:18:18.432319 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/jinja2/loaders.py", line 127, in load
[Wed Nov 04 21:18:18.432330 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     code = environment.compile(source, name, filename)
[Wed Nov 04 21:18:18.432341 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/jinja2/environment.py", line 638, in compile
[Wed Nov 04 21:18:18.432352 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     self.handle_exception(source=source_hint)
[Wed Nov 04 21:18:18.432365 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
[Wed Nov 04 21:18:18.436987 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     reraise(*rewrite_traceback_stack(source=source))
[Wed Nov 04 21:18:18.437013 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/usr/local/python3.7/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
[Wed Nov 04 21:18:18.437031 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     raise value.with_traceback(tb)
[Wed Nov 04 21:18:18.437047 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]   File "/home/mabass/public_html/flask-live/templates/adx_abctrainer/word.html", line 43, in template
[Wed Nov 04 21:18:18.437064 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568]     {{ form_2.max(min=min_abs, max=max_abs, class='form-control adx-form-int', value=max }}
[Wed Nov 04 21:18:18.437081 2020] [wsgi:error] [pid 26420] [client 1.2.3.4:58568] jinja2.exceptions.TemplateSyntaxError: unexpected '}', expected ')'

Maybe fix that curly brace to a parenthesis like it's expecting and see if it works? I censored out your IP, but quickly let me know if you need anything else censored from that log excerpt.

[mabass]

Thanks, though I think I fixed that specific issue shortly after. I'll have a closer look tomorrow.

 

Anyway, thanks for taking your time. This is amazing support.

[mabass]

Doing more testing, and it def. seems to be tied to server load.

Right now Tommy's at 14.14 load (Edit: now at 68) and the form's break reliably, with response times around 1s.

 

I tried pre-loading some image files to cache them user-side but that doesn't seem to help, the response delay is still too high.

 

Have you tried switchting to the httpd.conf method? I don't know enough about Apache/wsgi to tell if this could result in a speed gain.

 

 

Edit:

More testing. I'm observing a correlation with htaccess and related domains. If I reduce my ht-access to the bare minium, the waiting delays seem to improve.

 

Edited November 5, 2020 by mabass

[Krydos]

I renamed your .htaccess to disable it and set your mabass.tk domain up with a httpd.conf wsgi configuration but it still does the same thing. Sometimes it works and sometimes it doesn't.