Welcome
Welcome to refracta

You are currently viewing our boards as a guest, which gives you limited access to view most discussions and access our other features. By joining our free community, you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content, and access many other special features. In addition, registered members also see less advertisements. Registration is fast, simple, and absolutely free, so please, join our community today!

again a bash script question

If it's not on-topic, it's in here.

again a bash script question

Postby nadir » Sun Nov 13, 2016 8:45 am

What i do is this:
if lynx -dump -nolist some-http-webpage | grep "$TERM"
then
echo "found $TERM at some-http-webpage
fi

What i would like is to
store the result of lynx -dump in a file
and
if the file is elder than a week, use lynx again
else
grep through the file

thanks.
Last edited by nadir on Tue Nov 15, 2016 3:57 pm, edited 1 time in total.
So i herd u liek mudkip?
User avatar
nadir
 
Posts: 1160
Joined: Wed Mar 09, 2011 4:18 am
Location: here

Re: again a basch script question

Postby fsmithred » Sun Nov 13, 2016 9:03 pm

Redirect the wget output to file, then grep the file for $TERM

Before you do that, use 'find -mtime <whatever a week ago looks like> -name same-file-as-above' and based on the result, either grep the file or wget it and grep it. You can use mtime to find files up to some time in the past or beyond some time in the past. Been awhile since we done this, huh?
User avatar
fsmithred
 
Posts: 2095
Joined: Wed Mar 09, 2011 9:13 pm

Re: again a basch script question

Postby nadir » Mon Nov 14, 2016 7:14 am

Lol. Yes, i do remember something with times. But to me anything scripting related has been a while.

In the meantime i ran in a different problem.
This does work
Code: Select all
if lynx -dump -nolist http://main.mepis-deb.org/MX14packages.html | grep "$search_term"
then
FOUND="yes"
printf "$search_term found in community  repo http://main.mepis-deb.org/MX14packages.html \n"
fi

But as it's several repos, i put them in a list, and the result doesn't work anymore:
Code: Select all
code]
repo_list=(
        http://teharris.net/NewPackages.html
        http://main.mepis-deb.org/MX15packages.html
        http://main.mepis-deb.org/MX14packages.html
        )


#----------------------------------------------------------

for i in ${repo_list[*]}
do
#       if echo "$i"
        if lynx -dump -nolist "$i" | grep "$searchterm"
        then
                FOUND="yes"
                printf "\n$searchterm found in $i\n"
        fi
done

Doesn't work mean:
a) it puts all output of lynx on the screen, not only a sucessful grep and
b) it always says "found at ", no matter what

Here are the two complete scripts:
working way:
http://sprunge.us/COIH
not working with array and loop
http://sprunge.us/HYKT

Usage is: sh repo_crawler <searchterm>
So i herd u liek mudkip?
User avatar
nadir
 
Posts: 1160
Joined: Wed Mar 09, 2011 4:18 am
Location: here

Re: again a basch script question

Postby nadir » Mon Nov 14, 2016 8:23 pm

I found the problem:
1) variabe declared as search_term
search_term="$1"
and used as searchterm
2)
if lynx -dump -nolist "$i" | grep "$searchterm"

... now back to the file-date voodoo ...
So i herd u liek mudkip?
User avatar
nadir
 
Posts: 1160
Joined: Wed Mar 09, 2011 4:18 am
Location: here

Re: again a basch script question

Postby fsmithred » Mon Nov 14, 2016 10:35 pm

Your github page has been taken over by someone else. I was going to find an example that we worked on together. Anyway...
Code: Select all
# find files in current directory modified more than 30 days ago:
find . -type f -mtime +30

# find files in current directory modified in the last 10 days:
find . -type f -mtime -10


Code: Select all
if find . -name myfile -mtime +7 ; then
    if lynx -blah > myfile ; then
        grep whatever
        print whatever
   fi
else
    grep whatever
    print whatever
fi
User avatar
fsmithred
 
Posts: 2095
Joined: Wed Mar 09, 2011 9:13 pm

Re: again a basch script question

Postby nadir » Tue Nov 15, 2016 10:22 am

I got problems with it. Might be me.
But i assume that
a) find . -name dragora_repos.txt -mtime +7
will indeed not find dragoras_repos.txt
in case it is elder than a week
but
b) using if find ... ; will return a true value anyway.

Code: Select all
user$ find . -name dragora_repos.txt  -mtime +7
user$ echo $?
0
user$ rm dragora_repos.txt
user$ touch dragora_repos.txt
user$ if find . -name dragora_repos.txt -mtime +7; then echo "elder than a week"; fi
elder than a week
user$
So i herd u liek mudkip?
User avatar
nadir
 
Posts: 1160
Joined: Wed Mar 09, 2011 4:18 am
Location: here

Re: again a basch script question

Postby nadir » Tue Nov 15, 2016 11:08 am

i did it like this:
if [ $(find . -name "$log_file" -mtime +7) ]

The resulting codeblock is:
Code: Select all
#!/usr/bin/bash
### Trying to implement a text file to crawl_mepis.sh
### so we don't have to download all repos each time we run it
### **** knows if this works reliable
### hence it's actual tmp-name is: hack_crawl_mepis :-)

# 1) make sure search_term is given as argument
if [ "$#" -ne 1  ]
then
   printf "\nEnter program you search for as argument\n" 1>&2
   exit 1
fi
#----------------------------------------------------------
# 2) set variables
search_term="$1"
found=""
log_file="mepis_repos.txt"
#shorter URL list for testing purposes
repo_list=(     http://teharris.net/NewPackages.html
      http://main.mepis-deb.org/MX15packages.html
      http://main.mepis-deb.org/MX14packages.html )


# 3) if this is the first run, mepis_repo.txt doesn't exist
# run lynx to downlad
if [ ! -f "$log_file" ]
then

   for i in "${repo_list[@]}"
   do
      if lynx -dump -nolist "$i" | tee -a "$log_file" | grep "$search_term"
      then
         found="yes"   
         printf "\n%s found in $i\n" "$search_term"
      fi
   done
# if mepis_repos.txt exists, check it's been last updated
# if elder than x days, lynx dump the repos again
# else just grep through the file
else
   if   [ $(find . -name "$log_file"  -mtime +7) ]
   then
      rm "$log_file"
      for i in "${repo_list[@]}"
           do
                   if lynx -dump -nolist "$i" | tee -a "$log_file" | grep "$search_term"
                   then
                           found="yes"
                           printf "\n%s found in $i\n" "$search_term"
                   fi
           done
   else
      grep "$search_term" "$log_file"
   fi
fi



http://sprunge.us/TZhg

Will need more testing,
but thanks a lot, fsmithred.
So i herd u liek mudkip?
User avatar
nadir
 
Posts: 1160
Joined: Wed Mar 09, 2011 4:18 am
Location: here

Re: again a basch script question

Postby fsmithred » Tue Nov 15, 2016 3:37 pm

Yeah, I got the same problem and came up with the same solution

But I don't understand why you need the script. Doesn't 'aptitude search <package>' work in mepis?
.
User avatar
fsmithred
 
Posts: 2095
Joined: Wed Mar 09, 2011 9:13 pm

Re: again a basch script question

Postby nadir » Tue Nov 15, 2016 3:53 pm

glad to hear you came up with the same solution.

As far aptitude question is concerned:
I guess so. I don't use it.
The script is for dragora, there are community repos, right now you can't search them with the packagemanager.
Those are ca 6 or 7. If i need a package, i check via Web if they exist. Go to 1 url, use firefox'es find, go to the next URL, use firefoxes find, etc. That get's very timeconsuming, if i do it, say, 5 times a day (in the same time i already packaged it myself ... ).

Why did i use Mepis community repos instead? I am not sure how much impact the server of dragora can handle. And, another reason, i thought here more people might be interested in debian-based URL's (i assumed aptitude would not be able to handle the community Mepis repos).
So i herd u liek mudkip?
User avatar
nadir
 
Posts: 1160
Joined: Wed Mar 09, 2011 4:18 am
Location: here


Return to General Nonsense

Who is online

Users browsing this forum: No registered users and 0 guests

suspicion-preferred