Bazsalanszky/destreamer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

v2.0 RELEASE

Browse source
This commit is contained in:
snobu 2020-04-18 15:34:26 +03:00
parent 65847cb29d
commit 609cf43ee0
10 changed files with 125 additions and 114 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

151
README.md
View file

@ -1,134 +1,99 @@
# Destreamer
<a href="https://github.com/snobu/destreamer/actions">
<img src="https://github.com/snobu/destreamer/workflows/Node%20CI/badge.svg" alt="CI build status" />
</a>
![](logo.png)
![destreamer](assets/logo.png)
## Saves Microsoft Stream videos for offline enjoyment
Alpha-quality, don't expect much. It does work though, so that's a neat feature.
## v2.0 Release, codename _Hammer of Dawn<sup>TM</sup>_
It's slow (e.g. a 60-min video takes 20-30 minutes to download). Not much i can do about it for now unless i find a better way than ripping HLS.
This release would not have been possible without the code and time contributed by two distinguished developers: [@lukaarma](https://github.com/lukaarma) and [@kylon](https://github.com/kylon). Thank you!
## NEWS
## News
- We now have a token cache so we can reuse access tokens for their one hour lifetime. What this really means is that within one hour you only need to login via the popup browser once.
- We now have a token cache so we can reuse access tokens. This really means that within one hour you need to perform the interactive browser login only once.
- We removed the dependency on `youtube-dl`.
- Getting to the HLS URL is dramatically more reliable as we dropped parsing the DOM for the video element in favor of calling the Microsoft Stream API
## This project is now looking for contributors
<img src="https://www.whitesourcesoftware.com/wp-content/uploads/2018/02/10-github-to-follow.jpg" width=400 />
Roadmap -
- [X] Token cache (so you don't have to log in every time you run destreamer)
- [ ] Download closed captions if available
- [ ] Performance improvements (via aria2c maybe?) // _This is under consideration, we're not sure if this borders on abusing the streaming endpoints or not._
- [x] Single static binary (for each major OS)
Send a quality PR first and i'll add you as a contributor to the repository.
## DISCLAIMER
## Disclaimer
Hopefully this doesn't break the end user agreement for Microsoft Stream. Since we're simply saving the HLS stream to disk as if we were a browser, this does not abuse the streaming endpoints. However i take no responsibility if either Microsoft or your Office 365 admins request a chat with you in a small white room.
## PREREQS
## Prereqs
* **Node.js**: anything above v8.0 seems to work. A GitHub Action runs tests on all major Node versions on every commit.
* **ffmpeg**: a recent version (year 2019 or above), in `$PATH` or in the same directory as `destreamer.ts`.
- **Node.js**: anything above v8.0 seems to work. A GitHub Action runs tests on all major Node versions on every commit.
- **npm**: Usually comes with Node.js, type `npm` in your terminal to check for its presence
- **ffmpeg**: a recent version (year 2019 or above), in `$PATH` or in the same directory as this README file (project root).
Destreamer takes a [honeybadger](https://www.youtube.com/watch?v=4r7wHMg5Yjg) approach towards the OS it's running on, tested on Windows, macOS and Linux, results may vary, feel free to open an issue if trouble arise.
Destreamer takes a [honeybadger](https://www.youtube.com/watch?v=4r7wHMg5Yjg) approach towards the OS it's running on. We've successfully tested it on Windows, macOS and Linux.
## How to build
## HOW TO BUILD
You can build destreamer on any OS.
To build destreamer run the following commands, in order -
- `npm install`
- `npm run -s build`
You will find destreamer.js in `build/src` folder.
## Usage
To build destreamer.js run the following commands:
* `npm install`
* `npm run -s build`
## USAGE
* Unpack destreamer and chromium into the same folder
* Open a new terminal and navigate to that folder
* Run destreamer executable
Linux / MacOS
```
$ ./destreamer
```
Windows
```
destreamer.exe
```
## Options
```
$ node ./destreamer.js
$ ./destreamer.sh
Options:
--help Show help [boolean]
--version Show version number [boolean]
--username, -u [string]
--outputDirectory, -o [string] [default: "videos"]
--videoUrls, -V List of video urls or path to txt file containing the urls
[array] [required]
--simulate, -s Disable video download and print metadata
information to the console
[boolean] [default: false]
--noThumbnails, --nthumb Do not display video thumbnails
[boolean] [default: false]
--verbose, -v Print additional information to the console
(use this before opening an issue on GitHub)
[boolean] [default: false]
--help Show help [boolean]
--version Show version number [boolean]
--videoUrls, -i List of video urls [array]
--videoUrlsFile, -f Path to txt file containing the urls [string]
--username, -u [string]
--outputDirectory, -o The directory where destreamer will save your
downloads [default: videos] [string]
--outputDirectories, -O Path to a txt file containing one output directory
per video [string]
--noExperiments, -x Do not attempt to render video thumbnails in the
console [boolean] [default: false]
--simulate, -s Disable video download and print metadata information
to the console [boolean] [default: false]
--verbose, -v Print additional information to the console (use this
before opening an issue on GitHub)
[boolean] [default: false]
```
Make sure you use the right escape char for your shell if using line breaks.
Make sure you use the right script (`.sh`, `.ps1` or `.cmd`) and escape char (if using line breaks) for your shell.
PowerShell uses a backtick [ **`** ] and cmd.exe uses a caret [ **^** ].
Bash &mdash;
```bash
./destreamer.sh --username username@example.com --outputDirectory "videos" \
--videoUrls "https://web.microsoftstream.com/video/VIDEO-1" \
"https://web.microsoftstream.com/video/VIDEO-2"
Download a video -
```sh
$ ./destreamer.sh -i "https://web.microsoftstream.com/video/VIDEO-1"
```
PowerShell &mdash;
```powershell
.\destreamer.ps1 --username username@example.com --outputDirectory "videos" `
--videoUrls "https://web.microsoftstream.com/video/VIDEO-1" `
"https://web.microsoftstream.com/video/VIDEO-2"
Download a video to a custom path -
```sh
$ ./destreamer.sh -i "https://web.microsoftstream.com/video/VIDEO-1" -o /Users/hacker/Downloads
```
cmd.exe &mdash;
```cmd
destreamer --username username@example.com --outputDirectory "videos" ^
--videoUrls "https://web.microsoftstream.com/video/VIDEO-1" ^
"https://web.microsoftstream.com/video/VIDEO-2"
Download two or more videos -
```sh
$ ./destreamer.sh -i "https://web.microsoftstream.com/video/VIDEO-1" \
"https://web.microsoftstream.com/video/VIDEO-2"
```
You can create a `.txt` file containing your video URLs, one video per line. The text file can have any name, followed by the `.txt` extension. Run destreamer as follows:
```
./destreamer.sh --username username@example.com --outputDirectory "videos" \
--videoUrlsFile list.txt
Download many videos but read URLs from a file -
```sh
$ ./destreame.sh -f list.txt
```
You can create a `.txt` file containing your video URLs, one video per line. The text file can have any name, followed by the `.txt` extension.
Passing `--username` is optional. It's there to make logging in faster (the username field will be populated automatically on the login form).
You can use an absolute path for `--outputDirectory`, for example `/mnt/videos`.
You can use an absolute path for `-o` (output directory), for example `/mnt/videos`.
## RANDOM NOTE
## Expected output
Just ignore this error, we already have what we need to start the download, no time to deal with collaterals -
![screenshot](assets/screenshot-win.png)
![image](https://user-images.githubusercontent.com/6472374/77905069-4c585000-728e-11ea-914e-26f1ce5e595b.png)
By default, downloads are saved under `videos/` unless specified by `-o` (output directory).
## Found a bug?
## EXPECTED OUTPUT
```
<<<< OUTPUT >>>>
```
The video is now saved under `videos/`, or whatever the `outputDirectory` const points to.
Please open an [issue](https://github.com/snobu/destreamer/issues) and we'll look into it.

BIN
assets/logo.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 34 KiB

BIN
assets/screenshot-win.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 210 KiB

BIN
logo.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 70 KiB

20
src/CommandLineParser.ts
View file

@ -5,43 +5,51 @@ import colors from 'colors';
import fs from 'fs';
export const argv = yargs.options({
url: {
videoUrls: {
alias: 'i',
describe: 'List of video urls',
type: 'array',
demandOption: false
},
'from-file': {
videoUrlsFile: {
alias: 'f',
describe: 'Path to txt file containing the urls',
type: 'string',
demandOption: false
},
username: {
alias: 'u',
type: 'string',
demandOption: false
},
outdir: {
outputDirectory: {
alias: 'o',
describe: 'The directory where destreamer will save your downloads [default: videos]',
type: 'string',
demandOption: false
},
'out-dirs-from-file': {
outputDirectories: {
alias: 'O',
describe: 'Path to a txt file containing one output directory per video',
type: 'string',
demandOption: false
},
'no-experiments': {
describe: `Disable experimental features (do not display video thumbnails)`,
noExperiments: {
alias: 'x',
describe: `Do not attempt to render video thumbnails in the console`,
type: 'boolean',
default: false,
demandOption: false
},
simulate: {
alias: 's',
describe: `Disable video download and print metadata information to the console`,
type: 'boolean',
default: false,
demandOption: false
},
verbose: {
alias: 'v',
describe: `Print additional information to the console (use this before opening an issue on GitHub)`,
type: 'boolean',
default: false,

6
src/Errors.ts
View file

@ -48,13 +48,13 @@ export const enum CLI_ERROR {
GRACEFULLY_STOP = ' ', // gracefully stop execution, yargs way
MISSING_REQUIRED_ARG = 'You must specify a URLs source.\n' +
'Valid options are --videoUrls or --videoUrlsFile.',
'Valid options are -i for one or more URLs separated by sapce or -f for URLs from file.',
VIDEOURLS_ARG_CONFLICT = 'Too many URLs sources specified!\n' +
'Please specify a single URLs source with either --videoUrls or --videoUrlsFile.',
'Please specify a single source, either -i or -f (URLs from file)',
OUTPUTDIR_ARG_CONFLICT = 'Too many output arguments specified!\n' +
'Please specify a single output argument with either --outputDirectory or --outputDirectories.',
'Please specify a single output argument, either -o or --outputDirectories.',
FILE_INPUT_VIDEOURLS_ARG = 'Wrong input for option --videoUrls.\n' +
'To read URLs from file, use --videoUrlsFile option.',

25
src/TokenCache.ts
View file

@ -2,6 +2,8 @@ import * as fs from 'fs';
import { Session } from './Types';
import { bgGreen, bgYellow, green } from 'colors';
import jwtDecode from 'jwt-decode';
import axios from 'axios';
import colors from 'colors';
export class TokenCache {
private tokenCacheFile: string = '.token_cache';
@ -53,4 +55,27 @@ export class TokenCache {
console.info(green('Fresh access token dropped into .token_cache'));
});
}
public async RefreshToken(session: Session): Promise<string | null> {
let endpoint = `${session.ApiGatewayUri}refreshtoken?api-version=${session.ApiGatewayVersion}`;
let response = await axios.get(endpoint,
{
headers: {
Authorization: `Bearer ${session.AccessToken}`
}
});
let freshCookie: string | null = null;
try {
let cookie: string = response.headers["set-cookie"].toString();
freshCookie = cookie.split(',Authorization_Api=')[0];
}
catch (e) {
console.error(colors.yellow("Error when calling /refreshtoken: Missing or unexpected set-cookie header."));
}
return freshCookie;
}
}

1
src/Types.ts
View file

@ -4,7 +4,6 @@ export type Session = {
ApiGatewayVersion: string;
}
export type Metadata = {
date: string;
totalChunks: number; // Abstraction of FFmpeg timemark

3
src/Utils.ts
View file

@ -136,6 +136,7 @@ export function ffmpegTimemarkToChunk(timemark: string) {
const hrs = parseInt(timeVals[0]);
const mins = parseInt(timeVals[1]);
const secs = parseInt(timeVals[2]);
const chunk = (hrs * 60) + mins + (secs / 60);
return (hrs * 60) + mins + (secs / 60);
return chunk;
}

33
src/destreamer.ts
View file

@ -163,17 +163,34 @@ async function downloadVideo(videoUrls: string[], outputDirectories: string[], s
video.title = makeUniqueTitle(sanitize(video.title) + ' - ' + video.date, outputDirectories[j]);
// Very experimental inline thumbnail rendering
if (!argv.noThumbnails)
if (!argv.noExperiments)
await drawThumbnail(video.posterImage, session.AccessToken);
console.info('Spawning ffmpeg with access token and HLS URL. This may take a few seconds...\n');
console.info('Spawning ffmpeg with access token and HLS URL. This may take a few seconds...');
// Try to get a fresh cookie, else gracefully fall back
// to our session access token (Bearer)
let freshCookie = await tokenCache.RefreshToken(session);
let headers = `Authorization:\ Bearer\ ${session.AccessToken}`;
if (freshCookie) {
console.info(colors.green('Using a fresh cookie.'));
headers = `Cookie:\ ${freshCookie}`;
}
const outputPath = outputDirectories[j] + path.sep + video.title + '.mp4';
const ffmpegInpt = new FFmpegInput(video.playbackUrl, new Map([
['headers', `Authorization:\ Bearer\ ${session.AccessToken}`]
['headers', headers]
]));
const ffmpegOutput = new FFmpegOutput(outputPath);
const ffmpegCmd = new FFmpegCommand();
const cleanupFn = function () {
pbar.stop();
try {
fs.unlinkSync(outputPath);
} catch(e) {}
}
pbar.start(video.totalChunks, 0, {
speed: '0'
@ -203,13 +220,7 @@ async function downloadVideo(videoUrls: string[], outputDirectories: string[], s
process.exit(ERROR_CODE.UNK_FFMPEG_ERROR);
});
process.on('SIGINT', () => {
pbar.stop();
try {
fs.unlinkSync(outputPath);
} catch (e) {}
});
process.on('SIGINT', cleanupFn);
// let the magic begin...
await new Promise((resolve: any, reject: any) => {
@ -221,6 +232,8 @@ async function downloadVideo(videoUrls: string[], outputDirectories: string[], s
ffmpegCmd.spawn();
});
process.off('SIGINT', cleanupFn);
}
}