What is Mixamo Downloader
Mixamo Downloader is a project which can help organize 3D model character animations generated by Mixamo.com website.
Mixamo.com is a website where you can animate 3d characters and download the file containing the animation, for free. The website has 2484 animations, and it could take forever to download all animations for a character by hand. Mixamo Downloader will download all animations, store them in Google Cloud storage bucket and expose REST WebServices for manage them.
The saved animations can be further used for implementing character animation in a video game, as seen in the video below.
Mixamo Downloader features:
- is exposing WebServices for:
- User management:
- registration
- authentication
- Character management:
- sync characters from Mixamo Website to applications DB
- get list of characters saved in DB
- Animation management:
- sync animations from Mixamo Website to applications DB
- get list of animations saved in DB
- CharacterAnimation management:
- export character animation from Mixamo Website
- download exported character animations and save them in Google Cloud Storage
- get list of saved characters animation
- User management:
- The WebServices will be called by an Angular2 Application
Project implementation details:
- The application is implemented using Spring Boot Suite:
- spring-boot-starter-web running Jetty for Rest WebServices
- spring-boot-starter-data-jpa for managing data using Hibernate and MySql
- spring-boot-starter-security for securing the application (the application is secured using JWT)
Project Structure:
Bitbucket Repository -> https://bitbucket.org/deviscool/mixamo-backend
- Mixamo Downloader is composed of 2 subprojects:
- MixamoDownloaderUtil
- is a library project, here is implemented the main logic, the library can be used further for implementing WebServices or Worker Applications
- MixamoDownloaderWebServices
- in this project are implemented the REST WebServices using MixamoDownloaderUtil library
- MixamoDownloaderUtil
Code Structure:
MixamoDownloaderUtil
- cool.devis.mixamo.util.entity
- in this package are present the JPA Entity Classes
User.java
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
package cool.devis.mixamo.entity; import com.fasterxml.jackson.annotation.JsonProperty; import javax.persistence.Column; import javax.persistence.Entity; import javax.persistence.GeneratedValue; import javax.persistence.GenerationType; import javax.persistence.Id; import javax.persistence.Transient; import javax.validation.constraints.Size; import org.hibernate.validator.constraints.Email; import org.hibernate.validator.constraints.NotEmpty; /** * * @author eusebiu */ @Entity public class User { @Id @GeneratedValue(strategy = GenerationType.AUTO) @Column(name = "id") private int id; @Column(name = "email", nullable = false, unique = true) @Email(message = "Please provide a valid e-mail") @NotEmpty(message = "Please provide an e-mail") @JsonProperty("email") private String email; @Column(name = "password") @Transient @Size(min = 6,max = 255,message = "Your password must be between 6 and 255 characters") private String password; @Column(name = "first_name") @NotEmpty(message = "Please provide your first name") private String firstName; @Column(name = "last_name") @NotEmpty(message = "Please provide your last name") private String lastName; @Column(name = "enabled") private boolean enabled; @Column(name = "confirmation_token") private String confirmationToken; public String getConfirmationToken() { return confirmationToken; } public void setConfirmationToken(String confirmationToken) { this.confirmationToken = confirmationToken; } public int getId() { return id; } public void setId(int id) { this.id = id; } public String getPassword() { return password; } public void setPassword(String password) { this.password = password; } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } public String getEmail() { return email; } public void setEmail(String email) { this.email = email; } public boolean getEnabled() { return enabled; } public void setEnabled(boolean value) { this.enabled = value; } } |
- cool.devis.mixamo.util.repository
- in this package are present the JPA Repository Interfaces
UserRepository.java
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
package cool.devis.mixamo.util.repository; import cool.devis.mixamo.util.entity.Product; import org.springframework.data.domain.Page; import org.springframework.data.domain.Pageable; import org.springframework.data.jpa.repository.JpaRepository; /** * * @author Eusebiu */ public interface AnimationRepository extends JpaRepository<Product, Integer> { public Product findByUuid(String uuid); public Page<Product> findByType(String uuid,Pageable pageable); } |
- cool.devis.mixamo.util.service
- in this package are present the Service Classes
CharacterAnimationService.java
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 |
@Service public class CharacterAnimationService { @Autowired CharacterService characterService; @Autowired AnimationService animationService; @Autowired CharacterAnimationRepository characterAnimationRepository; @Value("${mixamo.api.key}") private String apiKey; public Page<CharacterAnimation> getCharacterAnimations(Pagination pagination) { PageRequest pageRequest = new PageRequest(pagination.getPage() - 1, pagination.getLimit()); Page<CharacterAnimation> characterAnimations = characterAnimationRepository.findAll(pageRequest); return characterAnimations; } public ExportDetails getAnimationDetailsForExport(String characterId, String animationId, String authorizationJWT) { String url = "https://www.mixamo.com/api/v1/products/" + animationId + "?similar=0&character_id=" + characterId; RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new HttpHeaders(); headers.set("X-Api-Key", apiKey); headers.setContentType(MediaType.APPLICATION_JSON); headers.set("authorization", authorizationJWT); HttpEntity<String> entity = new HttpEntity<>("", headers); ResponseEntity<ExportDetails> exchange = restTemplate.exchange(url, HttpMethod.GET, entity, ExportDetails.class); return exchange.getBody(); } public JobStatus exportPrimaryCharacterAnimation(int animationId, String authorizationJWT) throws JsonProcessingException { PrimaryCharacter primaryCharacter = characterService.getPrimaryCharacter(); Product animation = animationService.findById(animationId); JobStatus jobStatus = exportCharacterAnimation(animation, primaryCharacter.getPrimaryCharacterId(), authorizationJWT); Product character = characterService.findByUuid(primaryCharacter.getPrimaryCharacterId()); if (character == null) { character = savePrimaryCharacter(primaryCharacter); } save(animation, character, jobStatus); return jobStatus; } private void save(Product animation, Product character, JobStatus jobStatus) { CharacterAnimation characterAnimation = new CharacterAnimation(); characterAnimation.setAnimation(animation); characterAnimation.setCharacter(character); characterAnimation.setJobId(jobStatus.getStatus()); characterAnimation.setJobStatus(jobStatus.getStatus()); characterAnimationRepository.save(characterAnimation); } private Product savePrimaryCharacter(PrimaryCharacter primaryCharacter) { Product character = new Product(); character.seUuId(primaryCharacter.getPrimaryCharacterId()); character.setName(primaryCharacter.getPrimaryCharacterName()); character.setSource(primaryCharacter.getPrimaryCharacterSource()); character = characterService.save(character); return character; } public JobStatus exportCharacterAnimation(Product animation, String characterUuid, String authorizationJWT) throws JsonProcessingException { String url = "https://www.mixamo.com/api/v1/animations/export"; String json = buildExportRequest(characterUuid, animation, authorizationJWT); RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new HttpHeaders(); headers.set("X-Api-Key", apiKey); headers.setContentType(MediaType.APPLICATION_JSON); headers.set("authorization", authorizationJWT); HttpEntity<String> entity = new HttpEntity<>(json, headers); ResponseEntity<JobStatus> exchange = restTemplate.exchange(url, HttpMethod.POST, entity, JobStatus.class); return exchange.getBody(); } private String buildExportRequest(String characterUuid, Product animation, String authorizationJWT) throws JsonProcessingException { ExportRequest request = new ExportRequest(); ObjectMapper mapper = new ObjectMapper(); ExportDetails exportDetails = this.getAnimationDetailsForExport(characterUuid, animation.getUuId(), authorizationJWT); List gmsHashList = new ArrayList(); gmsHashList.add(exportDetails.getDetails().getGmsHash()); request.setGmsHash(gmsHashList); request.setCharacterId(characterUuid); request.setProductName(animation.getName()); request.setPreferences(getExportPreferences()); String json = mapper.writeValueAsString(request); return json; } private ExportPreferences getExportPreferences() { ExportPreferences preferences = new ExportPreferences(); preferences.setFormat("fbx7"); preferences.setFps("30"); preferences.setSkin("true"); preferences.setReducekf("0"); return preferences; } public JobStatus getJobStatus(String jobId, String authorizationJWT) { String url = "https://www.mixamo.com/api/v1/characters/" + jobId + "/monitor"; RestTemplate restTemplate = new RestTemplate(); HttpHeaders headers = new HttpHeaders(); headers.set("X-Api-Key", apiKey); headers.setContentType(MediaType.APPLICATION_JSON); headers.set("authorization", authorizationJWT); HttpEntity<String> entity = new HttpEntity<>(headers); ResponseEntity<JobStatus> exchange = restTemplate.exchange(url, HttpMethod.GET, entity, JobStatus.class); return exchange.getBody(); } public void downloadExportedFile(String url) throws MalformedURLException, IOException { File destination = new File("c://test//test.fbx"); FileUtils.copyURLToFile(new URL(url), destination); } } |
- cool.devis.mixamo.util.google
- in this package is present the GoogleStorageUtil class which contains methods for manage google cloud bucket files
GoogleStorageUtil.java
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 |
public final class GoogleStorageUtil { static final Logger LOG = (Logger) LoggerFactory.getLogger(GoogleStorageUtil.class); HttpTransport HTTP_TRANSPORT = new NetHttpTransport(); JsonFactory JSON_FACTORY = new JacksonFactory(); private GoogleCredential googleCredential = null; private Storage googleStorage = null; @Value("${google.storage.key}") String GOOGLE_STORAGE_KEY; @Value("${google.storage.email}") String GOOGLE_STORAGE_EMAIL; @Value("${google.storage.exponential.backoff.codes}") String EXPONENTIAL_BACKOFF_EXCEPTION_CODES; @Value("${google.storage.retry.count}") Integer GOOGLE_STORAGE_RETRY_COUNT; @Value("${google.exponential.backoff.miliseconds}") Integer MAXIMUM_BACKOFF_MILLISECONDS; @Value("${temp.folder}") String TEMP_FOLDER_PATH; public GoogleStorageUtil() throws GeneralSecurityException, IOException { this.googleCredential = GetGoogleStorageCredentials(); this.googleStorage = getGoogleStorage(); } public GoogleCredential GetGoogleStorageCredentials() throws GeneralSecurityException, IOException { File privateKey = new File(GOOGLE_STORAGE_KEY); GoogleCredential credential = new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT) .setJsonFactory(JSON_FACTORY) .setServiceAccountId(GOOGLE_STORAGE_EMAIL) .setServiceAccountScopes( Collections.singleton("https://www.googleapis.com/auth/devstorage.full_control")) .setServiceAccountPrivateKeyFromP12File(privateKey) .build(); return credential; } public String GenerateGoogleStorageToken() throws GeneralSecurityException, IOException, Exception { GoogleCredential credentials = GetGoogleStorageCredentials(); boolean operationExecuted = false; Integer retryCount = 0; while (!operationExecuted) { try { credentials.refreshToken(); operationExecuted = true; return credentials.getAccessToken(); } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } return ""; } private Storage getGoogleStorage() throws GeneralSecurityException, IOException { HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport(); Storage storageService = new Storage.Builder(httpTransport, JSON_FACTORY, googleCredential).setApplicationName("test").build(); return storageService; } public List<StorageObject> getStorageObjectListFromFolder(String bucketName, String folder) throws Exception { if (!folder.endsWith("/")) { folder = folder + "/"; } return GetStorageObjectList(bucketName, folder); } public List<StorageObject> GetStorageObjectList(String bucketName, String pathPrefix) throws IOException, Exception { boolean operationExecuted = false; Integer retryCount = 0; while (!operationExecuted) { try { Storage.Objects.List listRequest = googleStorage.objects().list(bucketName); listRequest.setPrefix(pathPrefix); List<StorageObject> results = new ArrayList<>(); Objects objects; // Iterate through each page of results, and add them to our results list. do { objects = listRequest.execute(); List<StorageObject> items = objects.getItems(); if (items != null) { // Add the items in this page of results to the list we'll return. results.addAll(items); } else { return results; } // Get the next page, in the next iteration of this loop. listRequest.setPageToken(objects.getNextPageToken()); } while (null != objects.getNextPageToken()); operationExecuted = true; return results; } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } return null; } public void downloadToOutputStream(String bucketName, String objectName, OutputStream data) throws IOException, InterruptedException, Exception { boolean operationExecuted = false; int retryCount = 0; while (!operationExecuted) { try { Storage.Objects.Get getObject = googleStorage.objects().get(bucketName, objectName); getObject.getMediaHttpDownloader().setDirectDownloadEnabled(false); getObject.executeMediaAndDownloadTo(data); operationExecuted = true; LOG.info("Download OK"); } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } } public void downloadFile(String bucketName, String objectName, String downloadPath) throws IOException, Exception { boolean operationExecuted = false; Integer retryCount = 0; FileOutputStream fileOut = null; while (!operationExecuted) { try { ByteArrayOutputStream out = new ByteArrayOutputStream(); downloadToOutputStream(bucketName, objectName, out); fileOut = new FileOutputStream(new File(downloadPath)); fileOut.write(out.toByteArray()); operationExecuted = true; LOG.info("Downloaded " + out.toByteArray().length + " bytes"); } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } finally { if (fileOut != null) { fileOut.close(); } } } } public byte[] downloadFile(String storagePath) throws IOException, Exception { ByteArrayOutputStream out = null; try { String bucketName = storagePath.split("/")[0]; String fileName = storagePath.replace(bucketName + "/", ""); out = new ByteArrayOutputStream(); downloadToOutputStream(bucketName, fileName, out); return out.toByteArray(); } finally { if (out != null) { out.close(); } } } public void moveFile(String sourceBucket, String sourceFile, String destinationBucket, String destinationFile) throws IOException, Exception { if (!sourceFile.equals(destinationFile)) { boolean operationExecuted = false; Integer retryCount = 0; while (!operationExecuted) { try { Storage.Objects.Copy copy = googleStorage.objects().copy(sourceBucket, sourceFile, destinationBucket, destinationFile, null); copy.execute(); Storage.Objects.Delete delete = googleStorage.objects().delete(sourceBucket, sourceFile); delete.execute(); operationExecuted = true; } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } } } public void moveFolder(String sourceBucket, String sourceFolder, String destinationBucket, String destinationFolder) throws IOException, Exception { if (!sourceFolder.endsWith("/")) { sourceFolder = sourceFolder + "/"; } if (!destinationFolder.endsWith("/")) { destinationFolder = destinationFolder + "/"; } List<StorageObject> sourceFiles = GetStorageObjectList(sourceBucket, sourceFolder); for (StorageObject sourceFile : sourceFiles) { moveFile(sourceBucket, sourceFile.getName(), destinationBucket, sourceFile.getName().replaceFirst(sourceFolder, destinationFolder)); } } public void deleteFolder(String sourceBucket, String sourceFolder) throws IOException, Exception { if (!sourceFolder.endsWith("/")) { sourceFolder = sourceFolder + "/"; } List<StorageObject> sourceFiles = GetStorageObjectList(sourceBucket, sourceFolder); for (StorageObject sourceFile : sourceFiles) { deleteFile(sourceBucket, sourceFile.getName()); } } public void copyFolder(String sourceBucket, String sourceFolder, String destinationBucket, String destinationFolder) throws IOException, Exception { List<StorageObject> sourceFiles = GetStorageObjectList(sourceBucket, sourceFolder + "/"); for (StorageObject sourceFile : sourceFiles) { copyFile(sourceBucket, sourceFile.getName(), destinationBucket, sourceFile.getName().replaceFirst(sourceFolder, destinationFolder)); } } public void copyFolder(String sourceBucket, String sourceFolder, String destinationBucket, String destinationFolder, HashMap extendedActions) throws IOException, Exception { List<StorageObject> sourceFiles = GetStorageObjectList(sourceBucket, sourceFolder + "/"); for (StorageObject sourceFile : sourceFiles) { if (extendedActions != null) { if (extendedActions.get("maximumLevel") != null) { int maximumLevel = (int) extendedActions.get("maximumLevel"); if (sourceFile.getName().split("/").length > maximumLevel) { continue; } } } copyFile(sourceBucket, sourceFile.getName(), destinationBucket, sourceFile.getName().replaceFirst(sourceFolder, destinationFolder)); } } public boolean checkIfFileExists(String bucket, String file) throws IOException, Exception { List<StorageObject> GetStorageObjectList = GetStorageObjectList(bucket, file); return GetStorageObjectList.stream().anyMatch((storageObject) -> (storageObject.getName().equals(file))); } public StorageObject getFile(String bucket, String filePath) throws IOException, Exception { List<StorageObject> storageObjectList = GetStorageObjectList(bucket, filePath); for (StorageObject storageObject : storageObjectList) { if (storageObject.getName().equals(filePath)) { return storageObject; } } return null; } public boolean checkIfFolderExists(String bucket, String folderName) throws IOException, Exception { List<StorageObject> GetStorageObjectList = GetStorageObjectList(bucket, folderName); return (GetStorageObjectList.size() > 0); } public StorageObject copyFile(String sourceBucket, String sourceFile, String destinationBucket, String destinationFile) throws IOException, Exception { Storage.Objects.Copy copy = googleStorage.objects().copy(sourceBucket, sourceFile, destinationBucket, destinationFile, null); StorageObject copiedObject = null; boolean operationExecuted = false; Integer retryCount = 0; while (!operationExecuted) { try { copiedObject = copy.execute(); operationExecuted = true; } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } String sourceFileName = FilenameUtils.getName(sourceFile); String sourceParentFolder = FilenameUtils.getPath(sourceFile); String destinationFileName = FilenameUtils.getName(destinationFile); String destinationParentFolder = FilenameUtils.getPath(destinationFile); return copiedObject; } public void deleteFile(String storageFilePath) throws IOException, Exception { String bucketName = storageFilePath.split("/")[0]; String fileName = storageFilePath.substring(storageFilePath.indexOf("/") + 1); deleteFile(bucketName, fileName); } public void deleteFile(String bucketName, String fileName) throws IOException, Exception { boolean operationExecuted = false; Integer retryCount = 0; while (!operationExecuted) { try { googleStorage.objects().delete(bucketName, fileName).execute(); operationExecuted = true; } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } } public String getFileHash(String bucketName, String filePath) throws Exception { MessageDigest digest = MessageDigest.getInstance("SHA-256"); ByteArrayOutputStream out = new ByteArrayOutputStream(); downloadToOutputStream(bucketName, filePath, out); ByteArrayInputStream is = new ByteArrayInputStream(out.toByteArray()); byte[] buffer = new byte[100 * 1024]; int read; String output = ""; try { while ((read = is.read(buffer)) > 0) { digest.update(buffer, 0, read); } byte[] sum = digest.digest(); //metodo 1 StringBuilder sb = new StringBuilder(); //metodo 2 for (int i = 0; i < sum.length; i++) { String hex = Integer.toHexString(0xff & sum[i]); if (hex.length() == 1) { sb.append('0'); } sb.append(hex); } output = sb.toString(); } catch (Exception ex) { throw ex; } finally { try { is.close(); } catch (Exception ex) { throw ex; } } return output; } public void uploadFile(String filePath, String bucketName, String uploadPath) throws FileNotFoundException, IOException, GeneralSecurityException, Exception { FileInputStream inputFile = new FileInputStream(new File(filePath)); String mimeType = getFileMIMETypeByExtension(uploadPath, false); uploadStream(uploadPath, mimeType, inputFile, bucketName); } public void uploadFile(byte[] fileByteArray, String bucketName, String uploadPath) throws FileNotFoundException, IOException, GeneralSecurityException, Exception { ByteArrayInputStream inputFile = new ByteArrayInputStream(fileByteArray); String mimeType = "application/octet-stream"; if (uploadPath.endsWith(".png")) { mimeType = "image/png"; } uploadStream(uploadPath, mimeType, inputFile, bucketName); } public void uploadFileWithEncryptionKey(String filePath, String bucketName, String uploadPath, String base64CseKey, String base64CseKeyHash) throws FileNotFoundException, IOException, GeneralSecurityException, Exception { FileInputStream inputFile = new FileInputStream(new File(filePath)); String mimeType = getFileMIMETypeByExtension(uploadPath, false); if (uploadPath.endsWith(".png")) { mimeType = "image/png"; } uploadStream(uploadPath, mimeType, inputFile, bucketName, base64CseKey, base64CseKeyHash); } /** * Uploads data to an object in a bucket. * * @param name the name of the destination object. * @param contentType the MIME type of the data. * @param stream the data - for instance, you can use a FileInputStream to * upload a file. * @param bucketName the name of the bucket to create the object in. * @throws java.io.IOException * @throws java.security.GeneralSecurityException */ private void uploadStream(String name, String contentType, InputStream stream, String bucketName) throws IOException, Exception { boolean operationExecuted = false; int retryCount = 0; while (!operationExecuted) { try { InputStreamContent contentStream = new InputStreamContent(contentType, stream); StorageObject objectMetadata = new StorageObject().setName(name); // Set the destination object name // Set the access control list to publicly read-only //.setAcl(Arrays.asList( // new ObjectAccessControll().setEntity("allUsers").setRole("READER"))); // Do the insert Storage.Objects.Insert insertRequest = googleStorage.objects().insert( bucketName, objectMetadata, contentStream); insertRequest.execute(); operationExecuted = true; LOG.info("upload OK"); } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } } public void uploadStream(String name, String contentType, InputStream stream, String bucketName, String base64CseKey, String base64CseKeyHash) throws IOException, Exception { boolean operationExecuted = false; int retryCount = 0; while (!operationExecuted) { try { InputStreamContent contentStream = new InputStreamContent(contentType, stream); StorageObject objectMetadata = new StorageObject().setName(name); // Set the destination object name // Set the access control list to publicly read-only //.setAcl(Arrays.asList( // new ObjectAccessControll().setEntity("allUsers").setRole("READER"))); // Do the insert Storage.Objects.Insert insertRequest = googleStorage.objects().insert( bucketName, objectMetadata, contentStream); insertRequest.getMediaHttpUploader().setDisableGZipContent(true); final HttpHeaders httpHeaders = new HttpHeaders(); httpHeaders.set("x-goog-encryption-algorithm", "AES256"); httpHeaders.set("x-goog-encryption-key", base64CseKey); httpHeaders.set("x-goog-encryption-key-sha256", base64CseKeyHash); insertRequest.setRequestHeaders(httpHeaders); insertRequest.execute(); operationExecuted = true; } catch (GoogleJsonResponseException ex) { LOG.info("Error uploading: " + ex.getContent()); LOG.error("Error uploading" + ex.getContent()); } catch (IOException ex) { retryCount = handleExponentialBackoff(ex, retryCount); LOG.error("Exception: ", ex); } } } public void zipFolder(String bucketName, String folderPath, String destinationBucketName, String zipPath) throws IOException, FileNotFoundException, GeneralSecurityException, Exception { zipFolder(bucketName, folderPath, destinationBucketName, zipPath, null); } public void zipFolder(File dir, File zipFile) throws IOException { FileOutputStream fout = new FileOutputStream(zipFile); ZipOutputStream zout = new ZipOutputStream(fout); zipSubFolder("", dir, zout); zout.close(); } private void zipSubFolder(String basePath, File dir, ZipOutputStream zout) throws IOException { byte[] buffer = new byte[4096]; File[] files = dir.listFiles(); for (File file : files) { if (file.isDirectory()) { String path = basePath + file.getName() + "/"; // zout.putNextEntry(new ZipEntry(path)); zipSubFolder(path, file, zout); zout.closeEntry(); } else { FileInputStream fin = new FileInputStream(file); zout.putNextEntry(new ZipEntry(basePath + file.getName())); int length; while ((length = fin.read(buffer)) > 0) { zout.write(buffer, 0, length); } zout.closeEntry(); fin.close(); } } } public void zipFolder(String bucketName, String folderPath, String destinationBucketName, String zipPath, HashMap extendedActions) throws FileNotFoundException, IOException, GeneralSecurityException, Exception { if (folderPath.endsWith("/")) { folderPath = folderPath.substring(0, folderPath.length() - 1); } List<StorageObject> items = getStorageObjectListFromFolder(bucketName, folderPath); String tempFolderPath = TEMP_FOLDER_PATH; String tempFile = tempFolderPath + "tmpFolder" + UUID.randomUUID().toString() + ".zip"; OutputStream out = new FileOutputStream(new File(tempFile)); ByteArrayOutputStream outByteStream = new ByteArrayOutputStream(); ZipOutputStream testZip = new ZipOutputStream(out); String baseName = FilenameUtils.getBaseName(folderPath); LOG.info("Base name: " + baseName); for (StorageObject storageObject : items) { if (extendedActions != null) { if (extendedActions.get("maximumLevel") != null) { int maximumLevel = (int) extendedActions.get("maximumLevel"); if (storageObject.getName().split("/").length > maximumLevel) { continue; } } } ZipEntry zipEntry = new ZipEntry(baseName + storageObject.getName().substring(folderPath.length())); ////???? downloadToOutputStream(bucketName, storageObject.getName(), outByteStream); testZip.putNextEntry(zipEntry); testZip.write(outByteStream.toByteArray()); outByteStream.flush(); outByteStream.reset(); testZip.closeEntry(); } outByteStream.close(); testZip.close(); uploadFile(tempFile, destinationBucketName, zipPath); new File(tempFile).delete(); } public void zipFiles(String bucketName, ArrayList<String> filePaths, String destinationBucketName, String zipPath, String baseName) throws FileNotFoundException, IOException, GeneralSecurityException, Exception { String tempFolderPath = TEMP_FOLDER_PATH; String tempFile = tempFolderPath + "temp" + UUID.randomUUID().toString() + ".zip"; OutputStream out = new FileOutputStream(new File(tempFile)); ByteArrayOutputStream outByteStream = new ByteArrayOutputStream(); ZipOutputStream testZip = new ZipOutputStream(out); //FilenameUtils.getBaseName(folderPath); LOG.info("Base name: " + baseName); for (String filePath : filePaths) { ZipEntry zipEntry = null; if (filePath.contains("AIP")) { zipEntry = new ZipEntry(baseName + filePath.substring(filePath.indexOf("AIP"))); } else { zipEntry = new ZipEntry(baseName + FilenameUtils.getName(filePath)); } downloadToOutputStream(bucketName, filePath, outByteStream); testZip.putNextEntry(zipEntry); testZip.write(outByteStream.toByteArray()); outByteStream.flush(); outByteStream.reset(); testZip.closeEntry(); } outByteStream.close(); testZip.close(); uploadFile(tempFile, destinationBucketName, zipPath); new File(tempFile).delete(); } public void zipFolders(String bucketName, ArrayList<String> folderPaths, String destinationBucketName, String zipPath, HashMap extendedActions) throws FileNotFoundException, IOException, GeneralSecurityException, Exception { String tempFolderPath = TEMP_FOLDER_PATH; String tempFile = tempFolderPath + "tmpFolder" + UUID.randomUUID().toString() + ".zip"; OutputStream out = new FileOutputStream(new File(tempFile)); ByteArrayOutputStream outByteStream = new ByteArrayOutputStream(); ZipOutputStream testZip = new ZipOutputStream(out); for (String folderPath : folderPaths) { if (folderPath.endsWith("/")) { folderPath = folderPath.substring(0, folderPath.length() - 1); } List<StorageObject> items = GetStorageObjectList(bucketName, folderPath); String baseName = FilenameUtils.getBaseName(folderPath); baseName = FilenameUtils.getBaseName(folderPath.substring(0, folderPath.lastIndexOf(baseName) - 1)) + "/" + baseName; if (extendedActions != null) { if (extendedActions.get("replaceFirstInBaseName") != null) { HashMap replaceFirst = (HashMap) extendedActions.get("replaceFirstInBaseName"); String toReplace = (String) replaceFirst.get("toReplace"); String replaceWith = (String) replaceFirst.get("replaceWith"); baseName = baseName.replaceFirst(toReplace, replaceWith); } } LOG.info("Base name: " + baseName); for (StorageObject storageObject : items) { if (extendedActions != null) { if (extendedActions.get("maximumLevel") != null) { int maximumLevel = (int) extendedActions.get("maximumLevel"); if (storageObject.getName().split("/").length > maximumLevel) { continue; } } } ZipEntry zipEntry = new ZipEntry(baseName + storageObject.getName().substring(folderPath.length())); downloadToOutputStream(bucketName, storageObject.getName(), outByteStream); testZip.putNextEntry(zipEntry); testZip.write(outByteStream.toByteArray()); outByteStream.flush(); outByteStream.reset(); testZip.closeEntry(); } } outByteStream.close(); testZip.close(); uploadFile(tempFile, destinationBucketName, zipPath); new File(tempFile).delete(); } public byte[] getStorageFileByteArray(String bucketName, String fileName) throws IOException, GeneralSecurityException, Exception { byte[] byteArray = null; ByteArrayOutputStream out = new ByteArrayOutputStream(); try { downloadToOutputStream(bucketName, fileName, out); byteArray = out.toByteArray(); } finally { out.close(); } return byteArray; } public String getFileMIMETypeByExtension(String fileName) throws Exception { return getFileMIMETypeByExtension(fileName, true); } public String getFileMIMETypeByExtension(String fileName, boolean throwExceptionIfNotRecognized) throws Exception { String MIMEType = ""; String extension = FilenameUtils.getExtension(fileName); switch (extension.toLowerCase()) { case "pdf": MIMEType = "application/pdf"; break; case "txt": MIMEType = "text/plain"; break; case "p7m": MIMEType = "application/pkcs7-mime"; break; case "tsd": MIMEType = "application/timestamped-data"; break; case "xml": MIMEType = "text/xml"; break; case "zip": MIMEType = "application/zip"; break; case "eml": MIMEType = "message/rfc822"; break; case "tiff": case "tif": MIMEType = "image/tiff"; break; case "jpeg": case "jpg": case "jpe": case "jfif": case "jfif-tbnl": MIMEType = "image/jpeg"; break; case "doc": MIMEType = "application/msword"; break; case "docx": MIMEType = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; break; case "xls": MIMEType = "application/excel"; break; case "xlsx": MIMEType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; break; case "ppt": MIMEType = "application/mspowerpoint"; break; case "pptx": MIMEType = "application/vnd.openxmlformats-officedocument.presentationml.presentation"; break; case "ods": MIMEType = "application/vnd.oasis.opendocument.spreadsheet"; break; case "odp": MIMEType = "application/vnd.oasis.opendocument.presentation"; break; case "odg": MIMEType = "application/vnd.oasis.opendocument.graphics"; break; case "odb": MIMEType = "application/vnd.oasis.opendocument.database"; break; case "png": MIMEType = "image/png"; break; } if (MIMEType.isEmpty()) { if (throwExceptionIfNotRecognized) { throw new Exception("MIME Type not recognized"); } else { MIMEType = "application/octet-stream"; } } return MIMEType; } public int getStatusCodeFromHttpResponseException(Exception ex) { int statusCode = 0; if (ex instanceof HttpResponseException) { HttpResponseException httpEx = (HttpResponseException) ex; statusCode = httpEx.getStatusCode(); LOG.info("Status Code: " + statusCode); } return statusCode; } public int getStatusCodeFromExceptionUsingPattern(Exception ex) { int statusCode = 0; Matcher matcher = Pattern.compile("^Server returned HTTP response code: (\\d+)").matcher(ex.getMessage()); if (matcher.find()) { statusCode = Integer.parseInt(matcher.group(1)); } return statusCode; } public int getStatusCode(Exception ex) { int statusCode = getStatusCodeFromHttpResponseException(ex); if (statusCode == 0) { statusCode = getStatusCodeFromExceptionUsingPattern(ex); } return statusCode; } public boolean checkIfExceptionNeedsToBeHandledByExponentialBackoff(int statusCode) { boolean useExponentialBackoff = false; String exceptionCodes = EXPONENTIAL_BACKOFF_EXCEPTION_CODES; if (exceptionCodes != null && !exceptionCodes.equals("")) { String[] exceptionCodesList = exceptionCodes.split(","); for (String exceptionCode : exceptionCodesList) { if (Integer.parseInt(exceptionCode.trim()) == statusCode) { useExponentialBackoff = true; break; } } } return useExponentialBackoff; } public int handleExponentialBackoff(Exception ex, int retryCount) throws Exception { int statusCode = this.getStatusCode(ex); if (statusCode != 0) { boolean useExponentialBackoff = checkIfExceptionNeedsToBeHandledByExponentialBackoff(statusCode); if (useExponentialBackoff) { if (++retryCount > GOOGLE_STORAGE_RETRY_COUNT) { throw ex; } long sleepTime = (long) Math.min(Math.pow(2, retryCount) * 1000 + Math.floor(Math.random() * 1000) + 1, MAXIMUM_BACKOFF_MILLISECONDS); Thread.sleep(sleepTime); return retryCount; } throw ex; } throw ex; } public StorageObject patch(String bucket, String filePath, StorageObject patch) throws IOException, IOException { Storage.Objects.Patch patch1 = googleStorage.objects().patch(bucket, filePath, patch); return patch1.execute(); } } |
- cool.devis.mixamo.util.pojo
- in this package are present the POJO classes used for calling Mixamo Website REST API, they were generated with the help of http://www.jsonschema2pojo.org after investigating the rest methods which are called by Mixamo Website using Google Chrome Network monitor
AnimationDetails.java
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 |
package cool.devis.mixamo.util.pojo; /** * * @author Eusebiu */ import java.util.HashMap; import java.util.Map; import com.fasterxml.jackson.annotation.JsonAnyGetter; import com.fasterxml.jackson.annotation.JsonAnySetter; import com.fasterxml.jackson.annotation.JsonIgnore; import com.fasterxml.jackson.annotation.JsonInclude; import com.fasterxml.jackson.annotation.JsonProperty; import com.fasterxml.jackson.annotation.JsonPropertyOrder; @JsonInclude(JsonInclude.Include.NON_NULL) @JsonPropertyOrder({ "supports_inplace", "loopable", "default_frame_length", "duration", "gms_hash" }) public class AnimationDetails { @JsonProperty("supports_inplace") private Boolean supportsInplace; @JsonProperty("loopable") private Boolean loopable; @JsonProperty("default_frame_length") private Integer defaultFrameLength; @JsonProperty("duration") private Double duration; @JsonProperty("gms_hash") private GmsHash gmsHash; @JsonIgnore private Map<String, Object> additionalProperties = new HashMap<String, Object>(); @JsonProperty("supports_inplace") public Boolean getSupportsInplace() { return supportsInplace; } @JsonProperty("supports_inplace") public void setSupportsInplace(Boolean supportsInplace) { this.supportsInplace = supportsInplace; } @JsonProperty("loopable") public Boolean getLoopable() { return loopable; } @JsonProperty("loopable") public void setLoopable(Boolean loopable) { this.loopable = loopable; } @JsonProperty("default_frame_length") public Integer getDefaultFrameLength() { return defaultFrameLength; } @JsonProperty("default_frame_length") public void setDefaultFrameLength(Integer defaultFrameLength) { this.defaultFrameLength = defaultFrameLength; } @JsonProperty("duration") public Double getDuration() { return duration; } @JsonProperty("duration") public void setDuration(Double duration) { this.duration = duration; } @JsonProperty("gms_hash") public GmsHash getGmsHash() { return gmsHash; } @JsonProperty("gms_hash") public void setGmsHash(GmsHash gmsHash) { this.gmsHash = gmsHash; } @JsonAnyGetter public Map<String, Object> getAdditionalProperties() { return this.additionalProperties; } @JsonAnySetter public void setAdditionalProperty(String name, Object value) { this.additionalProperties.put(name, value); } } |
- cool.devis.mixamo.util.common
- in this package we find some common class which is containing utility methods which can be used in other classes
MixamoDownloaderWebServices
- cool.devis.mixamo.webservice.Application.java
- is the main entry point of the application, here the Spring Boot Application will be initialized
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
package cool.devis.mixamo.webservice; import cool.devis.mixamo.util.repository.UserRepository; import org.springframework.boot.CommandLineRunner; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.boot.autoconfigure.domain.EntityScan; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.ComponentScan; import org.springframework.data.jpa.repository.config.EnableJpaRepositories; @SpringBootApplication @ComponentScan({"cool.devis.mixamo"}) @EntityScan("cool.devis.mixamo.util.entity") @EnableJpaRepositories("cool.devis.mixamo.util.repository") public class Application { /** * @param args the command line arguments */ public static void main(String[] args) { SpringApplication.run(Application.class, args); //read more https://spring.io/guides/gs/spring-boot/ } @Bean public CommandLineRunner init(UserRepository userRepository){ return (args)->{ }; } } |
- cool.devis.mixamo.webservice.controller
- in this package we have the REST Controller Class, methods present here will be exposed as REST WebServices
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
@RestController() @RequestMapping(path = "/character/animation") public class CharacterAnimationController { private static final Logger log = LoggerFactory.getLogger(CharacterAnimationController.class); String authorizationJWTMixamoSite = ""; @Autowired CharacterAnimationService characterAnimationService; @RequestMapping(method = RequestMethod.GET) public Page<CharacterAnimation> getCharacterAnimations(@RequestParam(name = "page", defaultValue = "1", required = false) int page, @RequestParam(name = "limit", defaultValue = "20", required = false) int limit) { Pagination pagination = new Pagination(); pagination.setPage(page); pagination.setLimit(limit); return characterAnimationService.getCharacterAnimations(pagination); } @RequestMapping(path = "/{animationId}", method = RequestMethod.POST) public JobStatus exportPrimaryCharacterAnimation(@PathVariable(name = "animationId") int animationId) throws Exception { JobStatus result = null; if (authorizationJWTMixamoSite.isEmpty()) { throw new Exception("authorizationJWTMixamoSite not set"); } try { result = characterAnimationService.exportPrimaryCharacterAnimation(animationId, authorizationJWTMixamoSite); } catch (Exception ex) { log.error(ex.getMessage()); throw ex; } return result; } @RequestMapping(path = "/{characterId}/{animationId}", method = RequestMethod.POST) public String export(@PathVariable(name = "characterId") int characterId, @PathVariable(name = "animationId") int animationId) { throw new NotImplementedException(); } } |
- cool.devis.mixamo.webservice.security
- in this package we have the classes used for configuring the Security; Application is secured using JWT Authentication
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
@Configuration @EnableWebSecurity public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http.csrf().disable().authorizeRequests() .antMatchers("/").permitAll() .antMatchers(HttpMethod.POST, "/login").permitAll() .anyRequest().authenticated() .and() // We filter the api/login requests .addFilterBefore(new JWTLoginFilter("/login", authenticationManager()), UsernamePasswordAuthenticationFilter.class) // And filter other requests to check the presence of JWT in header .addFilterBefore(new JWTAuthenticationFilter(), UsernamePasswordAuthenticationFilter.class); http.cors(); } @Bean public CorsConfigurationSource corsConfigurationSource() { final CorsConfiguration configuration = new CorsConfiguration(); configuration.setAllowedOrigins(Arrays.asList("*")); configuration.setAllowedMethods(Arrays.asList("HEAD", "GET", "POST", "PUT", "DELETE", "PATCH")); // setAllowCredentials(true) is important, otherwise: // The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. configuration.setAllowCredentials(true); // setAllowedHeaders is important! Without it, OPTIONS preflight request // will fail with 403 Invalid CORS request configuration.setAllowedHeaders(Arrays.asList("Authorization", "Cache-Control", "Content-Type")); final UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource(); source.registerCorsConfiguration("/**", configuration); return source; } @Value("${development}") private boolean developmentMode; @Autowired private UserDetailsServiceImp userDetailsService; @Bean public BCryptPasswordEncoder bCryptPasswordEncoder() { return new BCryptPasswordEncoder(); } @Override protected void configure(AuthenticationManagerBuilder auth) throws Exception { if (developmentMode) { auth.inMemoryAuthentication() .withUser("admin") .password("password") .roles("ADMIN"); } auth.userDetailsService(userDetailsService).passwordEncoder(new BCryptPasswordEncoder()); } } |
The Application is packaged in a single Capsule .jar as described here -> http://devis.cool/java/java-gradle-capsule-template-project/
To Do next:
- implement proper logging in every class
- implement exception handling mechanism
- write automated tests
- implement upload of exported character animation in Google Cloud
- implement download character animation method to be used by frontend
- implement Worker for exporting character animations