Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.97249539s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:47:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.972527462s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 484.949906ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:46:46 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 484.987297ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X...................X..........X.X...X....X....] duration: 3.5208152s trace: https://jaeger.dev.storj.io/trace/6b7b323e38fd7c13 FAIL (2025-12-08 23:46:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520836361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.827141986s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:46:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.827169118s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.089130445s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:46:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.089153188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.XX.........X..................X................X......X....] duration: 5.728060647s trace: https://jaeger.dev.storj.io/trace/2db4e0e9e1558615 FAIL (2025-12-08 23:46:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.728089201s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 483.808623ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:46:08 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 483.843068ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.814340847s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:46:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.814609269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 4.893363714s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:46:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.893392367s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............X...................X..........X.X...X....X...] duration: 3.52074065s trace: https://jaeger.dev.storj.io/trace/6b7b323e38fd7c13 FAIL (2025-12-08 23:45:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520765578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.665773742s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:45:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.665800954s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 477.631109ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:45:47 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 477.661537ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.000602483s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:45:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.00063263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..XX.........X..................X................X......X...] duration: 5.795418067s trace: https://jaeger.dev.storj.io/trace/2db4e0e9e1558615 FAIL (2025-12-08 23:45:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.795464925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.716716453s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:45:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.716734187s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 4.740257985s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:45:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.740292712s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.84512247s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.845181932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.08486926s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:44:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.084899438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............X...................X..........X.X...X....X..] duration: 3.4112766s trace: https://jaeger.dev.storj.io/trace/6b7b323e38fd7c13 FAIL (2025-12-08 23:44:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411366994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.899973037s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:44:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.900005799s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 474.014683ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:44:38 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.041915ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.751035486s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:44:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.751075563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.704517182s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:44:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.704546048s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.214540115s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:44:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.214568258s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.063870374s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:44:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.063894911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...XX.........X..................X................X......X..] duration: 5.727225694s trace: https://jaeger.dev.storj.io/trace/2db4e0e9e1558615 FAIL (2025-12-08 23:44:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.727240952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................X...................X..........X.X...X....X.] duration: 3.413589262s trace: https://jaeger.dev.storj.io/trace/6b7b323e38fd7c13 FAIL (2025-12-08 23:44:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.413616396s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 472.663781ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:43:58 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 472.689481ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.041904728s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:43:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.04193143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.971605929s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:43:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.97164295s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....XX.........X..................X................X......X.] duration: 5.83202998s trace: https://jaeger.dev.storj.io/trace/2db4e0e9e1558615 FAIL (2025-12-08 23:43:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.832048074s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 475.030667ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:43:35 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.060885ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X....XX.........X..................X................X......X] duration: 1m0.001592942s trace: https://jaeger.dev.storj.io/trace/2db4e0e9e1558615 FAIL (2025-12-08 23:43:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001687864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.................X...................X..........X.X...X....X] duration: 5.00057365s trace: https://jaeger.dev.storj.io/trace/6b7b323e38fd7c13 FAIL (2025-12-08 23:43:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000600561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.582918721s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:43:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.582946353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.731834593s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:43:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.731861674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.022984472s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:43:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.023016733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.916148538s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:43:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.916247576s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X...................X..........X.X...X....] duration: 3.41076606s trace: https://jaeger.dev.storj.io/trace/35c63dd154a0c40 FAIL (2025-12-08 23:43:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410796267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.661072608s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:42:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.661098168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.730097051s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:42:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.730120857s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 481.422018ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/28e057ab10cdba7a FAIL (2025-12-08 23:42:25 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 481.448148ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................X...................X..........X.X...X...] duration: 3.487318898s trace: https://jaeger.dev.storj.io/trace/35c63dd154a0c40 FAIL (2025-12-08 23:42:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487343996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.744173255s trace: https://jaeger.dev.storj.io/trace/70eff1f9f4e10b61 FAIL (2025-12-08 23:42:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528325775 (79h28m2.39940882s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528956506 (79h28m2.400039551s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.528973077 (79h28m2.400056122s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529417774 (79h28m2.400500819s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529436621 (79h28m2.400519666s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.529914309 (79h28m2.400997354s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530012517 (79h28m2.401095562s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530350450 (79h28m2.401433495s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.530519823 (79h28m2.401602868s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.531364691 (79h28m2.402447747s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532093319 (79h28m2.403176364s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532361340 (79h28m2.403444415s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532522386 (79h28m2.403605431s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532903311 (79h28m2.403986436s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.532933729 (79h28m2.404016764s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533388345 (79h28m2.40447139s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:02.533851798 (79h28m2.404934843s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Uploading piece... Dec 8 22:57:03.977738643 (79h28m3.848821688s) [upload=69419]: Next returned nil segment Dec 8 22:57:03.977751697 (79h28m3.848834742s) [upload=69419]: Waiting for error group managing segments... Dec 8 22:57:04.403387773 (79h28m4.274470808s) [noise=true , address=145.79.227.49:4356 , piece_id=O5DQKYNX7FDZRKIRY25RW3SZ5KDXCXKH4T6XUJ6IYKFC72GDCRMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.407353224 (79h28m4.278436269s) [noise=true , address=145.79.227.26:4688 , piece_id=IH62NH264R444YU2SO3WLQPYVXXRU6PKWCIZBTAYTKP47SMTRNBQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.409836543 (79h28m4.280919588s) [noise=true , address=145.79.227.31:4831 , piece_id=EVWDAPHQATUZJNJYPB62AYPD5HJOHNGC6I6MAGDJLTMSLDUEZSUA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.421208337 (79h28m4.292291372s) [noise=true , address=145.79.227.14:4872 , piece_id=RO3RUB3PK2ITTEH3LI2DFVWD35EOG4WN6HYANFH3IF54HZ6K5NVQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.427481372 (79h28m4.298564417s) [noise=true , address=145.79.227.32:4866 , piece_id=P5XZWSOR4GBSJNKYSUGQ226DCEMGO6BT2LHLDELTZ3BVYZFPGRYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.429023378 (79h28m4.300106413s) [noise=true , address=145.79.227.27:4712 , piece_id=BSFH4WLX2CIBF6IGQWHSFMZPTLKTY4HJYFYW4HWO5BX5K4ZDPPKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.450758486 (79h28m4.321841541s) [noise=true , address=77.90.31.17:4788 , piece_id=TURSVUGGN3TVBJNZ7SGTEXEXTOQUFPCXEJWXGZI4WQUCE6VF6JPA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.457596155 (79h28m4.3286792s) [noise=true , address=77.90.31.14:4680 , piece_id=PVQQJNFOTRRBDR6BB4ZX2XI3V5AVFS7SIA25PA7MGESQEBTOMPYA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.518520946 (79h28m4.389603992s) [noise=true , address=77.90.31.11:4580 , piece_id=2ZPAW6AALV6WCUNX2VSVEHHWA7E5HZ3GAUJ7G3CX5XQAUGJKVCMQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.519040065 (79h28m4.39012311s) [noise=true , address=45.91.116.13:4160 , piece_id=Z5CHBECHMQGUN6Z3AQ2JEF6ASSJXAAQP4QSCIE3F2LR6SC533K5A, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.522273622 (79h28m4.393356657s) [noise=true , address=77.90.31.15:4704 , piece_id=LXNLZCY4SED3RNYMEIWLOLUPEPWV2FGSU2QZQHZAJS6Y2FKN2OXQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.548112425 (79h28m4.41919547s) [noise=true , address=77.90.31.18:4812 , piece_id=FLYJS2XHJAUFNGRVZLSWID4OGO625NE252XQCVPXIMHB2JIWY3YQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.552539545 (79h28m4.42362258s) [noise=true , address=45.91.116.12:4242 , piece_id=R3RYY3CYZWADEOZFZ2T7CSXNJSNUW65SLLO3XN6ICV2CJLDKV52Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567383841 (79h28m4.438466876s) [noise=true , address=145.79.227.15:4878 , piece_id=X4C35IZSJX4U7CW4CMMDQEG54C3FC7XDBQE6QBKAREJY2YTIBS4Q, seg_pos={0 0}, upload=69419]: Done uploading piece. err: <nil> Dec 8 22:57:04.567410381 (79h28m4.438493426s) [seg_pos={0 0}, upload=69419]: Segment reached optimal threshold of 14 pieces. Dec 8 22:57:04.567661300 (79h28m4.438744335s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567670687 (79h28m4.438880394s) [noise=true , address=77.90.31.13:4657 , piece_id=XBTPY6OHXBCQ5BR7DCFJNGWM7ZYJLZKDSAT7D2NX6A577G3EZFKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.567892619 (79h28m4.438975695s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.567917978 (79h28m4.439001053s) [noise=true , address=77.90.31.19:4873 , piece_id=CJJS3LPDC5TEO7L67MJ6KZGME55IHPYIA34H335LQNEHAY2G6EWA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568120824 (79h28m4.439203869s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VQKskrBadKVwYpngNNeSJPn3knJs7WXtfz9D6syskCDz9EVSt): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568129731 (79h28m4.439212776s) [noise=true , address=45.91.116.6:4984 , piece_id=7OWOXURFJFGVALVCTEY5SGXFXT4XOFQW2DTJ7SVFBQHY5ELWLGKA, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568150291 (79h28m4.439233336s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Dec 8 22:57:04.568164968 (79h28m4.439248043s) [noise=true , address=77.90.31.16:4738 , piece_id=TUPXZKW53QUECWANCZZD7BTQ5HNAL7QSTSDO7TJXV5ZOJHURFKCQ, seg_pos={0 0}, upload=69419]: Long tail cancellation due to optimal threshold Dec 8 22:57:04.568190317 (79h28m4.439273352s) [seg_pos={0 0}, upload=69419]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:57:04.568316087 (79h28m4.439399122s) [upload=69419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:57:04.568323891 (79h28m4.439406936s) [upload=69419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:03:25.488072307 (79h34m25.359155352s) [upload=69419]: Done uploading segments... Dec 8 23:03:25.488194299 (79h34m25.359277344s) [upload=69419]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.744217329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.956489363s trace: https://jaeger.dev.storj.io/trace/4b02f3035ab8e89a FAIL (2025-12-08 23:42:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.956520191s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XX....XX.........X..................X................X......] duration: 5.962884979s trace: https://jaeger.dev.storj.io/trace/1b047a6260a57a4d FAIL (2025-12-08 23:42:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.962910798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.066121551s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:42:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.066151577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.089311448s trace: https://jaeger.dev.storj.io/trace/6a17c2831ae76153 FAIL (2025-12-08 23:41:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 23:22:58.638987633 (79h53m58.510070678s) [upload=69799]: Starting upload Dec 8 23:22:58.639070963 (79h53m58.510154008s) [upload=69799]: Done starting upload Dec 8 23:22:58.639157306 (79h53m58.510240352s) [upload=69799]: Uploading segments... Dec 8 23:22:58.639261395 (79h53m58.51034445s) [upload=69799]: Got next segment. Inline: true Dec 8 23:22:58.639269269 (79h53m58.510352315s) [upload=69799]: Waiting for error group managing segments... Dec 8 23:22:58.639355735 (79h53m58.51043878s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:22:58.639362948 (79h53m58.510446013s) [upload=69799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 8 23:22:58.639367427 (79h53m58.510450482s) [upload=69799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 23:23:57.956567374 (79h54m57.827650409s) [upload=69799]: Done uploading segments... Dec 8 23:23:57.956606829 (79h54m57.827689864s) [upload=69799]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.089344601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.73265303s trace: https://jaeger.dev.storj.io/trace/47d563c9d72817cd FAIL (2025-12-08 23:41:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 23:14:19.174026253 (79h45m19.045109298s) [upload=69675]: Starting upload Dec 8 23:14:19.174087710 (79h45m19.045170745s) [upload=69675]: Done starting upload Dec 8 23:14:19.174229249 (79h45m19.045312284s) [upload=69675]: Uploading segments... Dec 8 23:14:19.174339840 (79h45m19.045422885s) [upload=69675]: Got next segment. Inline: false Dec 8 23:14:19.174370498 (79h45m19.045453533s) [upload=69675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 23:14:19.174375207 (79h45m19.045458242s) [upload=69675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 23:14:19.413530548 (79h45m19.284613603s) [seg_pos={0 0}, upload=69675]: Begin upload segment... Dec 8 23:14:19.413603236 (79h45m19.284686281s) [seg_pos={0 0}, upload=69675]: Done begin upload segment. Dec 8 23:14:19.413635678 (79h45m19.284718713s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.413719387 (79h45m19.284802422s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414216194 (79h45m19.285299239s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414348725 (79h45m19.285431761s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.414714803 (79h45m19.285797838s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415001889 (79h45m19.286084924s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415254411 (79h45m19.286337456s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415542539 (79h45m19.286625574s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.415851578 (79h45m19.286934623s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.416185994 (79h45m19.287269029s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418253401 (79h45m19.289336447s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.418679503 (79h45m19.289762548s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419090485 (79h45m19.29017352s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419471040 (79h45m19.290554085s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.419857446 (79h45m19.290940491s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:19.420261004 (79h45m19.291344049s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Uploading piece... Dec 8 23:14:20.503447635 (79h45m20.37453067s) [noise=true , address=77.90.31.13:4626 , piece_id=TBJPJBYPBCQ4FDK7TY4B3K77USFPMI7FTHPGFD3JE5UQJVWFO7LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.077656063 (79h45m20.948739098s) [noise=true , address=77.90.31.19:4841 , piece_id=FVGRZMH2UCZKYLJVT4QI3D7W7JFMFONTCGCHXE4HQIJPTVAJECBQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 8 23:14:21.102087965 (79h45m20.973171s) [upload=69675]: Next returned nil segment Dec 8 23:14:21.102108795 (79h45m20.97319184s) [upload=69675]: Waiting for error group managing segments... Dec 8 23:14:21.387607813 (79h45m21.258690858s) [noise=true , address=145.79.227.21:4473 , piece_id=DM4B3J54XU74MJCN4Q22ULTOYITW2AK6BEWPALBZGAVV4JYRUSLQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397579922 (79h45m21.268662957s) [noise=true , address=145.79.227.25:4617 , piece_id=6LEBRVLCUJQMSB2H7WQMXDCYL7SAGMWRQAOI6YMQVTBHJRDJQ3HA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.397986636 (79h45m21.269069681s) [noise=true , address=145.79.227.27:4689 , piece_id=AHWSH2VXYRXFSGHCPVZGBIWM56563YPMDBS3PYE4VYOXYLLLR2LA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.404053740 (79h45m21.275136775s) [noise=true , address=145.79.227.41:4087 , piece_id=ASZJQYFJEG3HCDAJIM5PHCLVTX4JACB7JTF4XSPQKULII7ILIBYQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.406106347 (79h45m21.277189382s) [noise=true , address=145.79.227.16:4912 , piece_id=MMCYEPO52DMVN5OJ3NSQX5DAFC5HC46PJIJQCASF3SVXCOZPMCEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.413646555 (79h45m21.2847296s) [noise=true , address=145.79.227.26:4655 , piece_id=4CAYC363DD6VL35EWZVCFYETU3Z2L6RS3WE2S6CXP6YPZJDORYRA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.436334196 (79h45m21.307417241s) [noise=true , address=145.79.227.33:4870 , piece_id=H6RGYJTNW44NU5KHYHSDE2NAG6AFBLOLWATV5PRSHY3OUUGSRHOA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439477171 (79h45m21.310560206s) [noise=true , address=145.79.227.25:4619 , piece_id=5TI2WSM3RAAAHMPAAYFUXHQP4YE55SY3LJZ6IK5NON6QOZ72YVXA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.439992633 (79h45m21.311075678s) [noise=true , address=145.79.227.39:4015 , piece_id=O3IITQFRBB77EZ33CFHX6FST3DAMRUNJDZ2ZULJVRKZZCAS2SYWQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440383617 (79h45m21.311466662s) [noise=true , address=145.79.227.26:4653 , piece_id=NWM4E7SRKK5VKPZ26WYCSAIIGT2FXZHEXDQSNGDWVSQU3AHMUJVQ, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.440807935 (79h45m21.31189097s) [noise=true , address=145.79.227.27:4691 , piece_id=76JYJDSVKOMAQHYCJXOHNQDPBEAM6MFSCQZB2ZXF5L7QZPAZ2GBA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:21.441238014 (79h45m21.312321059s) [noise=true , address=145.79.227.25:4618 , piece_id=HRATAPUXZRLVTID7J4T3JRP3OW2PBWPROGHVH4UPZA4W45CMSJYA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: <nil> Dec 8 23:14:48.499366157 (79h45m48.370449213s) [noise=true , address=145.79.227.32:4834 , piece_id=REL3M4EWK4MFJKOOH3F6JBNKIZSF447FK6H7LRQL7NOKP2LFWT3A, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:44454->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499470807 (79h45m48.370553852s) [noise=true , address=145.79.227.38:4978 , piece_id=4RYTFTY74PEAZM2MSOEIXOQFABCTISOLB5NQ7T4BWZ7QQ3TCEEEA, seg_pos={0 0}, upload=69675]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:58040->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 23:14:48.499556110 (79h45m48.370639145s) [seg_pos={0 0}, upload=69675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 23:14:48.738524305 (79h45m48.60960734s) [upload=69675]: Done uploading segments... Dec 8 23:14:48.738564521 (79h45m48.609647566s) [upload=69675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.732682766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....................X...................X..........X.X...X..] duration: 3.398496045s trace: https://jaeger.dev.storj.io/trace/35c63dd154a0c40 FAIL (2025-12-08 23:41:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398539045s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)