Rework special link parsing in the post-processing of markup #3354

thehowl · 2018-01-11T21:03:31Z

Fixes #3327. Adds the ability to link emails in markdown, both [email protected], <[email protected]> and [hello](mailto:[email protected]), all of which previously didn't work. Also removed the autolink js library, as the link processing can be very easily done on the backend using a very simple blackfriday option.

Closes #3364. Closes #652.

In the new mechanism, the post processing is done by passing the input through a given set of functions, passed to a new postProcessCtx. This enables the ability to customise the post-processing features to enable, and thus also closes #2968. It also enables in the commit message many other features, such as linking, writing emails, and basically everything PostProcess does except for shortlinks.

Closes #3112 as well.

The only thing that seems to be needed is some unit tests, but this should be on its way for 1.5

thehowl · 2018-01-11T21:08:51Z

Oh, I guess I kinda forgot about the unit tests... will need to rewrite them

lafriks · 2018-01-11T21:37:56Z

But this will not render commit with just short sha clickable if I'm not wrong

thehowl · 2018-01-11T22:19:54Z

But this will not render commit with just short sha clickable if I'm not wrong

It should still do AFAIK? What do you mean?

I've placed [WIP] on the header for the moment because I'm too tired now to iron out the remaining issue, but it's mostly done

lafriks · 2018-01-12T07:37:04Z

I mean this (taken from tests):
You currently have:
<a href=\"http://localhost:3000/gogits/gogs/commit/b6dd6210eaebc915fd5be5579c58cce4da2e2579\" rel=\"nofollow\">http://localhost:3000/gogits/gogs/commit/b6dd6210eaebc915fd5be5579c58cce4da2e2579</a>
But it it needs to be (text between tag):
<a href=\"http://localhost:3000/gogits/gogs/commit/b6dd6210eaebc915fd5be5579c58cce4da2e2579\" rel=\"nofollow\">b6dd6210ea</a>

codecov-io · 2018-01-13T22:42:16Z

Codecov Report

Merging #3354 into master will increase coverage by 0.35%.
The diff coverage is 89.51%.

@@            Coverage Diff             @@
##           master    #3354      +/-   ##
==========================================
+ Coverage   35.79%   36.15%   +0.35%     
==========================================
  Files         285      285              
  Lines       40868    40905      +37     
==========================================
+ Hits        14630    14790     +160     
+ Misses      24068    23943     -125     
- Partials     2170     2172       +2

Impacted Files	Coverage Δ
modules/templates/helper.go	`47.91% <37.5%> (-1.75%)`	⬇️
modules/markup/markup.go	`67.79% <40%> (+0.52%)`	⬆️
modules/markup/markdown/markdown.go	`66.35% <92.3%> (-2.35%)`	⬇️
modules/markup/html.go	`89.44% <92.43%> (+35.14%)`	⬆️
models/repo_list.go	`65.62% <0%> (-1.57%)`	⬇️
modules/process/manager.go	`81.15% <0%> (+4.34%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 769ab1e...ab78354. Read the comment docs.

thehowl · 2018-01-13T22:44:22Z

Since I got sick of the state of html.go, I rewrote it entirely making it use an HTML tree instead of a tokenizer + a bunch of sprintfs. I took the chance also to rewrite a bunch of stuff. It just so happened that because this enabled much more modularity than before, it also meant that implementing #2968 was pretty trivial.

Note that this does not enable markdown on commits.

Reviews welcome.

lunny · 2018-01-14T07:42:21Z

@thehowl Thanks for you great PR. I think we have serval kinds of render scenes. Markup File on the tree, Wiki, Issues(comments) and Notification Mail body. Currently, all support markdown syntax and Gitea Postprocesses. Markup File supports other syntax and Gitea Postprocesses. But there are some differences between markup files and wikis. So a set of Post processes could be a group, markup files and wikis could be different group.

thehowl · 2018-01-14T08:37:36Z

You mean to make this into a different package?

ypnos · 2018-01-15T14:23:21Z

In Gitea 1.3.2, a URL like https://github.com/select2/select2/issues/4505#issuecomment-351836912 is not parsed correctly. The dash and the part after it are not seen as part of the URL.

Is this related to the work being done here?

thehowl · 2018-01-15T14:30:33Z

@ypnos Slightly related, however the link you posted doesn't look like a case we need to handle seeing as it is an URL for a GitHub issue, and not the local instance. On the local instance, that is something we already handle; see this: https://try.gitea.io/Howl/aaa/issues/1

ypnos · 2018-01-15T15:01:16Z

This is just an example of an external link with a dash in it. So external URLs in the format http://… instead of […](http://…) are not part of this PR then?

thehowl · 2018-01-15T16:44:48Z

External links in general are a part of this PR, though they were already supported before (using js, that is). They render as a simple link, e.g. https://google.com, instead of https://google.com. Is that what you mean?

ypnos · 2018-01-15T16:59:05Z

Great, so the current behavior is buggy and I wondered if you could check the new code for this. Please have a look at my example comment at https://try.gitea.io/Howl/aaa/issues/1

thehowl · 2018-01-15T17:26:21Z

Interesting. Then yes, the bug still exists in my code. The way this could be solved is by, instead of using a regex, to use strings.FieldsFunc and find in the resulting array the strings beginning with http:// or https:// and for which url.Parse doesn't return an error. But that seems too much to implement in this PR - which is already pretty big. I suggest you to create an issue, linking to this conversation.

As a workaround, I recommend you use the standard markdown feature, which is that of wrapping the URL in angle brackets (e.g. <https://google.com>).

ypnos · 2018-01-15T18:51:56Z

Thank you.

After some of my own research on regular expressions for matching URLs I conclude that it is not worth the trouble for achieving more proper URL detection, hence I will not file another issue on this.

Btw. I also believe the more complicated approach with url.Parse would not be the end to it, for example it would most probably include a period at the end of the URL, which should in general be excluded.

thehowl · 2018-02-20T15:18:23Z

rebased onto master. Please review.

lafriks

Other than these small issues otherwise it LG-TM

lafriks · 2018-02-20T19:29:29Z

modules/markup/html.go

+	}
+
+	// remove initial parts - because Render creates a whole HTML page.
+	const lenInit = len(`<html><head></head><body>`)


Can this instead use find <body> in string and respective closing tag to be more future proof?

lafriks · 2018-02-20T19:33:03Z

modules/markup/markup.go

@@ -69,7 +69,8 @@ func RenderWiki(filename string, rawBytes []byte, urlPrefix string, metas map[st
 func render(parser Parser, rawBytes []byte, urlPrefix string, metas map[string]string, isWiki bool) []byte {
 	urlPrefix = strings.Replace(urlPrefix, " ", "+", -1)
 	result := parser.Render(rawBytes, urlPrefix, metas, isWiki)
-	result = PostProcess(result, urlPrefix, metas, isWiki)
+	// TODO: one day the error should be checked and returned.
+	result, _ = PostProcess(result, urlPrefix, metas, isWiki)


Error should be at least logged

lafriks · 2018-02-20T19:34:01Z

modules/templates/helper.go

 	cleanMsg := template.HTMLEscapeString(msg)
-	fullMessage := string(markup.RenderIssueIndexPattern([]byte(cleanMsg), opts))
-	body := strings.Split(strings.TrimSpace(fullMessage), "\n")
+	fullMessage, _ := markup.RenderCommitMessage([]byte(cleanMsg), urlPrefix, "", metas)


lafriks · 2018-02-20T19:34:29Z

modules/templates/helper.go

-	msgLines := strings.Split(strings.TrimSpace(fullMessage), "\n")
+	// we can safely assume that it will not return any error, since there
+	// shouldn't be any special HTML.
+	fullMessage, _ := markup.RenderCommitMessage([]byte(cleanMsg), urlPrefix, urlDefault, metas)


Same here, log error

lafriks · 2018-02-20T19:39:16Z

modules/markup/html_internal_test.go

@@ -0,0 +1,378 @@
+package markup


Missing license comment

appleboy · 2018-02-21T02:08:28Z

modules/markup/html_internal_test.go

+
+	"code.gitea.io/gitea/modules/setting"
+	"code.gitea.io/gitea/modules/util"
+	"github.com/stretchr/testify/assert"


Missing empty line between local and external package.

See https://github.com/go-gitea/gitea/blob/master/CONTRIBUTING.md#styleguide

…er changes to processor logic

This is just a description to allow me to force push in order to restart the drone build.

tboerger added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jan 11, 2018

thehowl changed the title ~~Email links in markdown; autolinking from backend~~ [WIP] Email links in markdown; autolinking from backend Jan 11, 2018

lafriks added type/refactoring Existing code has been cleaned up. There should be no new functionality. type/enhancement An improvement of existing functionality pr/wip This PR is not ready for review labels Jan 11, 2018

thehowl mentioned this pull request Jan 13, 2018

URLs in markdown are rendered with Javascript #3364

Closed

1 task

thehowl changed the title ~~[WIP] Email links in markdown; autolinking from backend~~ [WIP] Rework special link parsing in the post-processing of markup Jan 13, 2018

thehowl changed the title ~~[WIP] Rework special link parsing in the post-processing of markup~~ Rework special link parsing in the post-processing of markup Jan 13, 2018

lunny added this to the 1.x.x milestone Jan 14, 2018

lafriks mentioned this pull request Jan 16, 2018

Dash in URL breaks hyperlink parsing #3095

Closed

thehowl mentioned this pull request Jan 16, 2018

Autolinking as specified by the GFM spec #3381

Closed

lunny mentioned this pull request Jan 16, 2018

Add autolinking extension for issue MD #3112

Closed

lafriks requested changes Feb 20, 2018

View reviewed changes

lafriks removed the pr/wip This PR is not ready for review label Feb 20, 2018

lafriks modified the milestones: 1.x.x, 1.5.0 Feb 20, 2018

lafriks approved these changes Feb 20, 2018

View reviewed changes

tboerger added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 20, 2018

appleboy suggested changes Feb 21, 2018

View reviewed changes

thehowl added 12 commits February 22, 2018 17:24

Get rid of autolink

38aca97

autolink in markdown

0c86b94

Replace email addresses with mailto links

4b2be0b

better handling of links

49c7156

Remove autolink.js from footer

f55b2d9

Refactor entire html.go

fcf05be

fix some bugs

e0a3ee2

Make tests green, move what we can to html_internal_test, various oth…

85f227f

…er changes to processor logic

Make markdown tests work again

dc9f467

This is just a description to allow me to force push in order to restart the drone build.

Fix failing markdown tests in routers/api/v1/misc

00060dd

Add license headers, log errors, future-proof <body>

746d28d

fix formatting

886d945

appleboy approved these changes Feb 27, 2018

View reviewed changes

tboerger added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Feb 27, 2018

Merge branch 'master' into fix-3327

ab78354

lafriks merged commit 535445c into go-gitea:master Feb 27, 2018

thehowl mentioned this pull request Mar 3, 2018

URLs in markdown are not auto linked #3446

Closed

7 tasks

thehowl deleted the fix-3327 branch April 22, 2018 20:02

SagePtr mentioned this pull request Sep 13, 2018

Markdown tables are rendered bellow the footer #4212

Closed

4 tasks

go-gitea locked and limited conversation to collaborators Nov 24, 2020

Uh oh!

Rework special link parsing in the post-processing of markup #3354

Rework special link parsing in the post-processing of markup #3354

Uh oh!

Conversation

thehowl commented Jan 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thehowl commented Jan 11, 2018

Uh oh!

lafriks commented Jan 11, 2018

Uh oh!

thehowl commented Jan 11, 2018

Uh oh!

lafriks commented Jan 12, 2018

Uh oh!

codecov-io commented Jan 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

thehowl commented Jan 13, 2018

Uh oh!

lunny commented Jan 14, 2018

Uh oh!

thehowl commented Jan 14, 2018

Uh oh!

ypnos commented Jan 15, 2018

Uh oh!

thehowl commented Jan 15, 2018

Uh oh!

ypnos commented Jan 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thehowl commented Jan 15, 2018

Uh oh!

ypnos commented Jan 15, 2018

Uh oh!

thehowl commented Jan 15, 2018

Uh oh!

ypnos commented Jan 15, 2018

Uh oh!

thehowl commented Feb 20, 2018

Uh oh!

lafriks left a comment

Choose a reason for hiding this comment

Uh oh!

lafriks Feb 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lafriks Feb 20, 2018

Choose a reason for hiding this comment

Uh oh!

lafriks Feb 20, 2018

Choose a reason for hiding this comment

Uh oh!

lafriks Feb 20, 2018

Choose a reason for hiding this comment

Uh oh!

lafriks Feb 20, 2018

Choose a reason for hiding this comment

Uh oh!

appleboy Feb 21, 2018

Choose a reason for hiding this comment

Uh oh!

thehowl Feb 22, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thehowl commented Jan 11, 2018 •

edited

Loading

codecov-io commented Jan 13, 2018 •

edited

Loading

ypnos commented Jan 15, 2018 •

edited

Loading

lafriks Feb 20, 2018 •

edited

Loading