Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..XX..X..X.........................XX.................] duration: 20.581527338s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:53:10 UTC) orbiter warning: context deadline exceeded Duration: 20.581592692s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.........................X...................X...........] duration: 23.032062986s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:53:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 23.032091569s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 11.451208134s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:53:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.45124309s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 38.864440897s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:52:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.864457939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 7.073419412s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:52:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.073451582s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.340143954s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:52:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.340187827s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXX] duration: 5.002019488s trace: https://jaeger.dev.storj.io/trace/33c735f15f3a8045 FAIL (2025-12-15 13:52:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002087377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 16.169659905s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:52:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.169681106s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..XX..X..X.........................XX................] duration: 15.26686679s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:51:59 UTC) orbiter warning: context deadline exceeded Duration: 15.266888261s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.........................X...................X..........] duration: 14.82685395s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:51:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 14.826882153s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 7.95581604s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:51:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.955867298s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXX] duration: 5.000978606s trace: https://jaeger.dev.storj.io/trace/13932a14161efac3 FAIL (2025-12-15 13:51:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001000047s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 8.79928458s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.799439213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..XX..X..X.........................XX...............] duration: 15.612791804s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:51:08 UTC) orbiter warning: context deadline exceeded Duration: 15.61282159s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [X.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXX] duration: 5.000763045s trace: https://jaeger.dev.storj.io/trace/495542725270ebe5 FAIL (2025-12-15 13:50:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000784526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.........................X...................X.........] duration: 9.97026281s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:50:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.970295813s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 11.506728598s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:50:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.506781629s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..XX..X..X.........................XX..............] duration: 25.46616755s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:50:34 UTC) orbiter warning: context deadline exceeded Duration: 25.466200293s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 19.77796146s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:50:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.777982211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.894568142s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:50:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.894592959s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.........................X...................X........] duration: 16.083745774s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:50:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.083775301s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 21.895746128s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:50:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.895775994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.X.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXX] duration: 5.001052888s trace: https://jaeger.dev.storj.io/trace/3d914d6a51906a49 FAIL (2025-12-15 13:49:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001078409s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X..XX..X..X.........................XX.............] duration: 20.208610394s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:49:38 UTC) orbiter warning: context deadline exceeded Duration: 20.208626063s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.........................X...................X.......] duration: 10.145537358s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:49:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.145566874s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.47989112s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:49:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.479924312s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X..XX..X..X.........................XX............] duration: 22.182628364s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:49:18 UTC) orbiter warning: context deadline exceeded Duration: 22.182659452s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 17.97005826s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:49:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.970087836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [..X.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XX] duration: 5.001124455s trace: https://jaeger.dev.storj.io/trace/27d089d4e660ccb7 FAIL (2025-12-15 13:49:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001177345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.819762959s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:48:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.819787896s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 17.312812609s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:48:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.312828901s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..........X..XX..X..X.........................XX...........] duration: 21.285683306s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:48:39 UTC) orbiter warning: context deadline exceeded Duration: 21.285733973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.........................X...................X......] duration: 10.964200937s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:48:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.964226234s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 17.056354519s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:48:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.056387452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X..X.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.X] duration: 5.000891366s trace: https://jaeger.dev.storj.io/trace/30542f342bceb345 FAIL (2025-12-15 13:48:00 UTC) orbiter warning: context deadline exceeded Duration: 5.000917444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..........X..XX..X..X.........................XX..........] duration: 18.0991567s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:47:39 UTC) orbiter warning: context deadline exceeded Duration: 18.09919883s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.X..X.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.] duration: 5.000662481s trace: https://jaeger.dev.storj.io/trace/7abb0029e60f3f91 FAIL (2025-12-15 13:47:38 UTC) orbiter warning: context deadline exceeded Duration: 5.000688532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.........................X...................X.....] duration: 9.208418036s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:47:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.208449948s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.268210239s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:47:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.268242492s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 19.083576036s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:47:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.08360384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.........................X...................X....] duration: 25.241650311s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:47:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 25.241683684s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..........X..XX..X..X.........................XX.........] duration: 27.915876242s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:46:47 UTC) orbiter warning: context deadline exceeded Duration: 27.915907481s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.X..X.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X] duration: 5.001046157s trace: https://jaeger.dev.storj.io/trace/7abb0029e60f3f91 FAIL (2025-12-15 13:46:47 UTC) orbiter warning: context deadline exceeded Duration: 5.001071076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 25.364050016s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:46:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.364068431s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 9.878983526s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:46:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.879005538s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X..........X..XX..X..X.........................XX........] duration: 16.098659232s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:46:16 UTC) orbiter warning: context deadline exceeded Duration: 16.098712482s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.........................X...................X...] duration: 9.331934424s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:46:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.331967027s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 9.279610488s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:45:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.279643804s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.X..X.XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....] duration: 4.251266356s trace: https://jaeger.dev.storj.io/trace/6b485aa1834ea948 FAIL (2025-12-15 13:45:35 UTC) orbiter warning: context deadline exceeded Duration: 4.251293968s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..X..........X..XX..X..X.........................XX.......] duration: 25.853096306s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:45:28 UTC) orbiter warning: context deadline exceeded Duration: 25.853138115s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)