Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.00377216s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:59:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.003862984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X...................X.............X..X..] duration: 18.345259993s trace: https://jaeger.dev.storj.io/trace/28fc23e26fd78efe FAIL (2025-12-15 13:59:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:57:40.906144191 (238h28m40.777227237s) [upload=209086]: Starting upload Dec 15 13:57:40.906213463 (238h28m40.777296498s) [upload=209086]: Done starting upload Dec 15 13:57:40.906268909 (238h28m40.777351944s) [upload=209086]: Uploading segments... Dec 15 13:57:40.906392504 (238h28m40.777475539s) [upload=209086]: Got next segment. Inline: false Dec 15 13:57:40.906421329 (238h28m40.777504364s) [upload=209086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:57:40.906427100 (238h28m40.777510135s) [upload=209086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:57:41.243776519 (238h28m41.114859564s) [seg_pos={0 0}, upload=209086]: Begin upload segment... Dec 15 13:57:41.243841442 (238h28m41.114924487s) [seg_pos={0 0}, upload=209086]: Done begin upload segment. Dec 15 13:57:41.243866801 (238h28m41.114949836s) [noise=true , address=145.79.227.49:4339 , piece_id=K7YAH3OTAXLFMTVNLPDZPIMVMFN63PYAQM6L5CE3QEMYLVODPJ6Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.243883312 (238h28m41.114966357s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.243952303 (238h28m41.115035338s) [noise=true , address=145.79.227.42:4121 , piece_id=NI5XQ5L54VRU4A5KZV3XKEOTUJJHAPB6DGC6OAHR27D3TUZ3ECIQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.244383204 (238h28m41.11546626s) [noise=true , address=145.79.227.37:4567 , piece_id=O5JGU3ARUT67336SDWYXWEMQI22B2QUSILZ723DRVYSC2JIRR2GA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.244837910 (238h28m41.115920945s) [noise=true , address=145.79.227.33:4869 , piece_id=E5BYHFZSNEHQRHR5KF3OPVD3OPSSBW6AXCYLMLMYUIHJ5ZLQLA5Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.245290873 (238h28m41.116373908s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.245841771 (238h28m41.116924807s) [noise=true , address=145.79.227.34:4905 , piece_id=GBZX2L5NEQGAI22ALAWRG63BGJTCW5RMOLYNRM4NOJJGF3KPYDNA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.246293010 (238h28m41.117376055s) [noise=true , address=45.91.116.6:4975 , piece_id=V3AEIFWSPMOXIJ6FXNOZYLGDJTD2UVKHWT5PAREKKD6772VFFASA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.246836055 (238h28m41.1179191s) [noise=true , address=145.79.227.35:4483 , piece_id=UUIJ4XBOMHBYT2DV6GAKQ3FYHZH3WC7IEZ32DZCZ6OTB43UEYEGA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.248580416 (238h28m41.119663461s) [noise=true , address=145.79.227.28:4725 , piece_id=DA3ATIM3ATNCHA6T2OIFKMGQJDNIRWGFPMQREPNJ2OYJ2NRVQFNQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.250505361 (238h28m41.121588396s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.252126839 (238h28m41.123209884s) [noise=true , address=145.79.227.46:4602 , piece_id=CZ3S6QYYUEH4LJZCDIG3XUIAC4ASOQ2ISYLUJGVZI6BVDUWL6K6A, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.253782983 (238h28m41.124866028s) [noise=true , address=77.90.31.11:4555 , piece_id=VD2OGDO3IJALUKNVJTVZVNDSVRZB24VYCVMX56GEN43ST4ERYTYA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.254169830 (238h28m41.125252875s) [noise=true , address=77.90.31.10:4518 , piece_id=ZLVYEDO5HYHUD4S6UW2RJFHKESXDARPIQ3PWCXHQCUZSWVA5P4PQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.255413938 (238h28m41.126496983s) [noise=true , address=145.79.227.47:4267 , piece_id=6ARBYHKQDJRJXMLR5MTSR2MHPSIDWTAQC47UDTZ5WPS34ETE7PWQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.256786602 (238h28m41.127869647s) [noise=true , address=145.79.227.43:4158 , piece_id=33Y7RBUBTIPRHR42CJ2V2VCNWPECOCVLIERK3J3PTBVSQCDMEI5Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:43.118573177 (238h28m42.989656212s) [upload=209086]: Next returned nil segment Dec 15 13:57:43.118587074 (238h28m42.989670119s) [upload=209086]: Waiting for error group managing segments... Dec 15 13:57:43.419440166 (238h28m43.290523201s) [noise=true , address=77.90.31.10:4518 , piece_id=ZLVYEDO5HYHUD4S6UW2RJFHKESXDARPIQ3PWCXHQCUZSWVA5P4PQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:43.477434979 (238h28m43.348518024s) [noise=true , address=77.90.31.11:4555 , piece_id=VD2OGDO3IJALUKNVJTVZVNDSVRZB24VYCVMX56GEN43ST4ERYTYA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.144299770 (238h28m44.015382805s) [noise=true , address=145.79.227.35:4483 , piece_id=UUIJ4XBOMHBYT2DV6GAKQ3FYHZH3WC7IEZ32DZCZ6OTB43UEYEGA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.154823399 (238h28m44.025906434s) [noise=true , address=145.79.227.42:4121 , piece_id=NI5XQ5L54VRU4A5KZV3XKEOTUJJHAPB6DGC6OAHR27D3TUZ3ECIQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.164568556 (238h28m44.035651591s) [noise=true , address=145.79.227.49:4339 , piece_id=K7YAH3OTAXLFMTVNLPDZPIMVMFN63PYAQM6L5CE3QEMYLVODPJ6Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.193415696 (238h28m44.064498731s) [noise=true , address=45.91.116.6:4975 , piece_id=V3AEIFWSPMOXIJ6FXNOZYLGDJTD2UVKHWT5PAREKKD6772VFFASA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.259141441 (238h28m44.130224476s) [noise=true , address=145.79.227.28:4725 , piece_id=DA3ATIM3ATNCHA6T2OIFKMGQJDNIRWGFPMQREPNJ2OYJ2NRVQFNQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.264248206 (238h28m44.135331241s) [noise=true , address=145.79.227.37:4567 , piece_id=O5JGU3ARUT67336SDWYXWEMQI22B2QUSILZ723DRVYSC2JIRR2GA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.284564552 (238h28m44.155647587s) [noise=true , address=145.79.227.33:4869 , piece_id=E5BYHFZSNEHQRHR5KF3OPVD3OPSSBW6AXCYLMLMYUIHJ5ZLQLA5Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.287854075 (238h28m44.15893711s) [noise=true , address=145.79.227.47:4267 , piece_id=6ARBYHKQDJRJXMLR5MTSR2MHPSIDWTAQC47UDTZ5WPS34ETE7PWQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.308163007 (238h28m44.179246043s) [noise=true , address=145.79.227.46:4602 , piece_id=CZ3S6QYYUEH4LJZCDIG3XUIAC4ASOQ2ISYLUJGVZI6BVDUWL6K6A, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.313178167 (238h28m44.184261212s) [noise=true , address=145.79.227.34:4905 , piece_id=GBZX2L5NEQGAI22ALAWRG63BGJTCW5RMOLYNRM4NOJJGF3KPYDNA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.316682811 (238h28m44.187765846s) [noise=true , address=145.79.227.43:4158 , piece_id=33Y7RBUBTIPRHR42CJ2V2VCNWPECOCVLIERK3J3PTBVSQCDMEI5Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.316703942 (238h28m44.187786977s) [seg_pos={0 0}, upload=209086]: Segment reached optimal threshold of 13 pieces. Dec 15 13:57:44.316904434 (238h28m44.187987479s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 15 13:57:44.316921846 (238h28m44.188004901s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317050902 (238h28m44.188133937s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 15 13:57:44.317069087 (238h28m44.188152132s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317407521 (238h28m44.188490566s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: context canceled; context canceled Dec 15 13:57:44.317417851 (238h28m44.188500897s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317442407 (238h28m44.188525453s) [seg_pos={0 0}, upload=209086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:57:44.317542859 (238h28m44.188625904s) [upload=209086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:57:44.317560463 (238h28m44.188643508s) [upload=209086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:57:44.761912425 (238h28m44.63299546s) [upload=209086]: Done uploading segments... Dec 15 13:57:44.761932923 (238h28m44.633015968s) [upload=209086]: Upload finished. err: <nil> Duration: 18.345285933s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..X..X.........................XX.......................X.X] duration: 30.001062091s trace: https://jaeger.dev.storj.io/trace/1d01a26a7b8910d1 FAIL (2025-12-15 13:59:30 UTC) orbiter warning: context deadline exceeded Duration: 30.00110377s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX...XXX.X] duration: 5.000995347s trace: https://jaeger.dev.storj.io/trace/18869916524f8c23 FAIL (2025-12-15 13:59:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00101795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 21.745645024s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:59:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.745701621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX..X..X.........................XX.......................X.] duration: 28.138682211s trace: https://jaeger.dev.storj.io/trace/6823c0a27978a50a FAIL (2025-12-15 13:58:54 UTC) orbiter warning: context deadline exceeded Duration: 28.138711266s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.149897074s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:58:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.149923865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...................X.............X..X.] duration: 11.546269865s trace: https://jaeger.dev.storj.io/trace/28fc23e26fd78efe FAIL (2025-12-15 13:58:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:57:40.906144191 (238h28m40.777227237s) [upload=209086]: Starting upload Dec 15 13:57:40.906213463 (238h28m40.777296498s) [upload=209086]: Done starting upload Dec 15 13:57:40.906268909 (238h28m40.777351944s) [upload=209086]: Uploading segments... Dec 15 13:57:40.906392504 (238h28m40.777475539s) [upload=209086]: Got next segment. Inline: false Dec 15 13:57:40.906421329 (238h28m40.777504364s) [upload=209086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:57:40.906427100 (238h28m40.777510135s) [upload=209086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:57:41.243776519 (238h28m41.114859564s) [seg_pos={0 0}, upload=209086]: Begin upload segment... Dec 15 13:57:41.243841442 (238h28m41.114924487s) [seg_pos={0 0}, upload=209086]: Done begin upload segment. Dec 15 13:57:41.243866801 (238h28m41.114949836s) [noise=true , address=145.79.227.49:4339 , piece_id=K7YAH3OTAXLFMTVNLPDZPIMVMFN63PYAQM6L5CE3QEMYLVODPJ6Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.243883312 (238h28m41.114966357s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.243952303 (238h28m41.115035338s) [noise=true , address=145.79.227.42:4121 , piece_id=NI5XQ5L54VRU4A5KZV3XKEOTUJJHAPB6DGC6OAHR27D3TUZ3ECIQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.244383204 (238h28m41.11546626s) [noise=true , address=145.79.227.37:4567 , piece_id=O5JGU3ARUT67336SDWYXWEMQI22B2QUSILZ723DRVYSC2JIRR2GA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.244837910 (238h28m41.115920945s) [noise=true , address=145.79.227.33:4869 , piece_id=E5BYHFZSNEHQRHR5KF3OPVD3OPSSBW6AXCYLMLMYUIHJ5ZLQLA5Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.245290873 (238h28m41.116373908s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.245841771 (238h28m41.116924807s) [noise=true , address=145.79.227.34:4905 , piece_id=GBZX2L5NEQGAI22ALAWRG63BGJTCW5RMOLYNRM4NOJJGF3KPYDNA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.246293010 (238h28m41.117376055s) [noise=true , address=45.91.116.6:4975 , piece_id=V3AEIFWSPMOXIJ6FXNOZYLGDJTD2UVKHWT5PAREKKD6772VFFASA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.246836055 (238h28m41.1179191s) [noise=true , address=145.79.227.35:4483 , piece_id=UUIJ4XBOMHBYT2DV6GAKQ3FYHZH3WC7IEZ32DZCZ6OTB43UEYEGA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.248580416 (238h28m41.119663461s) [noise=true , address=145.79.227.28:4725 , piece_id=DA3ATIM3ATNCHA6T2OIFKMGQJDNIRWGFPMQREPNJ2OYJ2NRVQFNQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.250505361 (238h28m41.121588396s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.252126839 (238h28m41.123209884s) [noise=true , address=145.79.227.46:4602 , piece_id=CZ3S6QYYUEH4LJZCDIG3XUIAC4ASOQ2ISYLUJGVZI6BVDUWL6K6A, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.253782983 (238h28m41.124866028s) [noise=true , address=77.90.31.11:4555 , piece_id=VD2OGDO3IJALUKNVJTVZVNDSVRZB24VYCVMX56GEN43ST4ERYTYA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.254169830 (238h28m41.125252875s) [noise=true , address=77.90.31.10:4518 , piece_id=ZLVYEDO5HYHUD4S6UW2RJFHKESXDARPIQ3PWCXHQCUZSWVA5P4PQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.255413938 (238h28m41.126496983s) [noise=true , address=145.79.227.47:4267 , piece_id=6ARBYHKQDJRJXMLR5MTSR2MHPSIDWTAQC47UDTZ5WPS34ETE7PWQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.256786602 (238h28m41.127869647s) [noise=true , address=145.79.227.43:4158 , piece_id=33Y7RBUBTIPRHR42CJ2V2VCNWPECOCVLIERK3J3PTBVSQCDMEI5Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:43.118573177 (238h28m42.989656212s) [upload=209086]: Next returned nil segment Dec 15 13:57:43.118587074 (238h28m42.989670119s) [upload=209086]: Waiting for error group managing segments... Dec 15 13:57:43.419440166 (238h28m43.290523201s) [noise=true , address=77.90.31.10:4518 , piece_id=ZLVYEDO5HYHUD4S6UW2RJFHKESXDARPIQ3PWCXHQCUZSWVA5P4PQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:43.477434979 (238h28m43.348518024s) [noise=true , address=77.90.31.11:4555 , piece_id=VD2OGDO3IJALUKNVJTVZVNDSVRZB24VYCVMX56GEN43ST4ERYTYA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.144299770 (238h28m44.015382805s) [noise=true , address=145.79.227.35:4483 , piece_id=UUIJ4XBOMHBYT2DV6GAKQ3FYHZH3WC7IEZ32DZCZ6OTB43UEYEGA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.154823399 (238h28m44.025906434s) [noise=true , address=145.79.227.42:4121 , piece_id=NI5XQ5L54VRU4A5KZV3XKEOTUJJHAPB6DGC6OAHR27D3TUZ3ECIQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.164568556 (238h28m44.035651591s) [noise=true , address=145.79.227.49:4339 , piece_id=K7YAH3OTAXLFMTVNLPDZPIMVMFN63PYAQM6L5CE3QEMYLVODPJ6Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.193415696 (238h28m44.064498731s) [noise=true , address=45.91.116.6:4975 , piece_id=V3AEIFWSPMOXIJ6FXNOZYLGDJTD2UVKHWT5PAREKKD6772VFFASA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.259141441 (238h28m44.130224476s) [noise=true , address=145.79.227.28:4725 , piece_id=DA3ATIM3ATNCHA6T2OIFKMGQJDNIRWGFPMQREPNJ2OYJ2NRVQFNQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.264248206 (238h28m44.135331241s) [noise=true , address=145.79.227.37:4567 , piece_id=O5JGU3ARUT67336SDWYXWEMQI22B2QUSILZ723DRVYSC2JIRR2GA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.284564552 (238h28m44.155647587s) [noise=true , address=145.79.227.33:4869 , piece_id=E5BYHFZSNEHQRHR5KF3OPVD3OPSSBW6AXCYLMLMYUIHJ5ZLQLA5Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.287854075 (238h28m44.15893711s) [noise=true , address=145.79.227.47:4267 , piece_id=6ARBYHKQDJRJXMLR5MTSR2MHPSIDWTAQC47UDTZ5WPS34ETE7PWQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.308163007 (238h28m44.179246043s) [noise=true , address=145.79.227.46:4602 , piece_id=CZ3S6QYYUEH4LJZCDIG3XUIAC4ASOQ2ISYLUJGVZI6BVDUWL6K6A, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.313178167 (238h28m44.184261212s) [noise=true , address=145.79.227.34:4905 , piece_id=GBZX2L5NEQGAI22ALAWRG63BGJTCW5RMOLYNRM4NOJJGF3KPYDNA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.316682811 (238h28m44.187765846s) [noise=true , address=145.79.227.43:4158 , piece_id=33Y7RBUBTIPRHR42CJ2V2VCNWPECOCVLIERK3J3PTBVSQCDMEI5Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.316703942 (238h28m44.187786977s) [seg_pos={0 0}, upload=209086]: Segment reached optimal threshold of 13 pieces. Dec 15 13:57:44.316904434 (238h28m44.187987479s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 15 13:57:44.316921846 (238h28m44.188004901s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317050902 (238h28m44.188133937s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 15 13:57:44.317069087 (238h28m44.188152132s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317407521 (238h28m44.188490566s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: context canceled; context canceled Dec 15 13:57:44.317417851 (238h28m44.188500897s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317442407 (238h28m44.188525453s) [seg_pos={0 0}, upload=209086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:57:44.317542859 (238h28m44.188625904s) [upload=209086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:57:44.317560463 (238h28m44.188643508s) [upload=209086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:57:44.761912425 (238h28m44.63299546s) [upload=209086]: Done uploading segments... Dec 15 13:57:44.761932923 (238h28m44.633015968s) [upload=209086]: Upload finished. err: <nil> Duration: 11.546300202s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 10.457257784s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:58:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.457287921s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX...XXX.] duration: 3.723316878s trace: https://jaeger.dev.storj.io/trace/1bf9d5a38f3e5c46 FAIL (2025-12-15 13:58:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.723345713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XX..X..X.........................XX.......................X] duration: 30.000913601s trace: https://jaeger.dev.storj.io/trace/6823c0a27978a50a FAIL (2025-12-15 13:58:20 UTC) orbiter warning: context deadline exceeded Duration: 30.000936253s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 12.78260523s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:58:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.782632352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X...................X.............X..X] duration: 30.002743491s trace: https://jaeger.dev.storj.io/trace/28fc23e26fd78efe FAIL (2025-12-15 13:58:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:57:40.906144191 (238h28m40.777227237s) [upload=209086]: Starting upload Dec 15 13:57:40.906213463 (238h28m40.777296498s) [upload=209086]: Done starting upload Dec 15 13:57:40.906268909 (238h28m40.777351944s) [upload=209086]: Uploading segments... Dec 15 13:57:40.906392504 (238h28m40.777475539s) [upload=209086]: Got next segment. Inline: false Dec 15 13:57:40.906421329 (238h28m40.777504364s) [upload=209086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:57:40.906427100 (238h28m40.777510135s) [upload=209086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:57:41.243776519 (238h28m41.114859564s) [seg_pos={0 0}, upload=209086]: Begin upload segment... Dec 15 13:57:41.243841442 (238h28m41.114924487s) [seg_pos={0 0}, upload=209086]: Done begin upload segment. Dec 15 13:57:41.243866801 (238h28m41.114949836s) [noise=true , address=145.79.227.49:4339 , piece_id=K7YAH3OTAXLFMTVNLPDZPIMVMFN63PYAQM6L5CE3QEMYLVODPJ6Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.243883312 (238h28m41.114966357s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.243952303 (238h28m41.115035338s) [noise=true , address=145.79.227.42:4121 , piece_id=NI5XQ5L54VRU4A5KZV3XKEOTUJJHAPB6DGC6OAHR27D3TUZ3ECIQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.244383204 (238h28m41.11546626s) [noise=true , address=145.79.227.37:4567 , piece_id=O5JGU3ARUT67336SDWYXWEMQI22B2QUSILZ723DRVYSC2JIRR2GA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.244837910 (238h28m41.115920945s) [noise=true , address=145.79.227.33:4869 , piece_id=E5BYHFZSNEHQRHR5KF3OPVD3OPSSBW6AXCYLMLMYUIHJ5ZLQLA5Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.245290873 (238h28m41.116373908s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.245841771 (238h28m41.116924807s) [noise=true , address=145.79.227.34:4905 , piece_id=GBZX2L5NEQGAI22ALAWRG63BGJTCW5RMOLYNRM4NOJJGF3KPYDNA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.246293010 (238h28m41.117376055s) [noise=true , address=45.91.116.6:4975 , piece_id=V3AEIFWSPMOXIJ6FXNOZYLGDJTD2UVKHWT5PAREKKD6772VFFASA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.246836055 (238h28m41.1179191s) [noise=true , address=145.79.227.35:4483 , piece_id=UUIJ4XBOMHBYT2DV6GAKQ3FYHZH3WC7IEZ32DZCZ6OTB43UEYEGA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.248580416 (238h28m41.119663461s) [noise=true , address=145.79.227.28:4725 , piece_id=DA3ATIM3ATNCHA6T2OIFKMGQJDNIRWGFPMQREPNJ2OYJ2NRVQFNQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.250505361 (238h28m41.121588396s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.252126839 (238h28m41.123209884s) [noise=true , address=145.79.227.46:4602 , piece_id=CZ3S6QYYUEH4LJZCDIG3XUIAC4ASOQ2ISYLUJGVZI6BVDUWL6K6A, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.253782983 (238h28m41.124866028s) [noise=true , address=77.90.31.11:4555 , piece_id=VD2OGDO3IJALUKNVJTVZVNDSVRZB24VYCVMX56GEN43ST4ERYTYA, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.254169830 (238h28m41.125252875s) [noise=true , address=77.90.31.10:4518 , piece_id=ZLVYEDO5HYHUD4S6UW2RJFHKESXDARPIQ3PWCXHQCUZSWVA5P4PQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.255413938 (238h28m41.126496983s) [noise=true , address=145.79.227.47:4267 , piece_id=6ARBYHKQDJRJXMLR5MTSR2MHPSIDWTAQC47UDTZ5WPS34ETE7PWQ, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:41.256786602 (238h28m41.127869647s) [noise=true , address=145.79.227.43:4158 , piece_id=33Y7RBUBTIPRHR42CJ2V2VCNWPECOCVLIERK3J3PTBVSQCDMEI5Q, seg_pos={0 0}, upload=209086]: Uploading piece... Dec 15 13:57:43.118573177 (238h28m42.989656212s) [upload=209086]: Next returned nil segment Dec 15 13:57:43.118587074 (238h28m42.989670119s) [upload=209086]: Waiting for error group managing segments... Dec 15 13:57:43.419440166 (238h28m43.290523201s) [noise=true , address=77.90.31.10:4518 , piece_id=ZLVYEDO5HYHUD4S6UW2RJFHKESXDARPIQ3PWCXHQCUZSWVA5P4PQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:43.477434979 (238h28m43.348518024s) [noise=true , address=77.90.31.11:4555 , piece_id=VD2OGDO3IJALUKNVJTVZVNDSVRZB24VYCVMX56GEN43ST4ERYTYA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.144299770 (238h28m44.015382805s) [noise=true , address=145.79.227.35:4483 , piece_id=UUIJ4XBOMHBYT2DV6GAKQ3FYHZH3WC7IEZ32DZCZ6OTB43UEYEGA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.154823399 (238h28m44.025906434s) [noise=true , address=145.79.227.42:4121 , piece_id=NI5XQ5L54VRU4A5KZV3XKEOTUJJHAPB6DGC6OAHR27D3TUZ3ECIQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.164568556 (238h28m44.035651591s) [noise=true , address=145.79.227.49:4339 , piece_id=K7YAH3OTAXLFMTVNLPDZPIMVMFN63PYAQM6L5CE3QEMYLVODPJ6Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.193415696 (238h28m44.064498731s) [noise=true , address=45.91.116.6:4975 , piece_id=V3AEIFWSPMOXIJ6FXNOZYLGDJTD2UVKHWT5PAREKKD6772VFFASA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.259141441 (238h28m44.130224476s) [noise=true , address=145.79.227.28:4725 , piece_id=DA3ATIM3ATNCHA6T2OIFKMGQJDNIRWGFPMQREPNJ2OYJ2NRVQFNQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.264248206 (238h28m44.135331241s) [noise=true , address=145.79.227.37:4567 , piece_id=O5JGU3ARUT67336SDWYXWEMQI22B2QUSILZ723DRVYSC2JIRR2GA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.284564552 (238h28m44.155647587s) [noise=true , address=145.79.227.33:4869 , piece_id=E5BYHFZSNEHQRHR5KF3OPVD3OPSSBW6AXCYLMLMYUIHJ5ZLQLA5Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.287854075 (238h28m44.15893711s) [noise=true , address=145.79.227.47:4267 , piece_id=6ARBYHKQDJRJXMLR5MTSR2MHPSIDWTAQC47UDTZ5WPS34ETE7PWQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.308163007 (238h28m44.179246043s) [noise=true , address=145.79.227.46:4602 , piece_id=CZ3S6QYYUEH4LJZCDIG3XUIAC4ASOQ2ISYLUJGVZI6BVDUWL6K6A, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.313178167 (238h28m44.184261212s) [noise=true , address=145.79.227.34:4905 , piece_id=GBZX2L5NEQGAI22ALAWRG63BGJTCW5RMOLYNRM4NOJJGF3KPYDNA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.316682811 (238h28m44.187765846s) [noise=true , address=145.79.227.43:4158 , piece_id=33Y7RBUBTIPRHR42CJ2V2VCNWPECOCVLIERK3J3PTBVSQCDMEI5Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: <nil> Dec 15 13:57:44.316703942 (238h28m44.187786977s) [seg_pos={0 0}, upload=209086]: Segment reached optimal threshold of 13 pieces. Dec 15 13:57:44.316904434 (238h28m44.187987479s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 15 13:57:44.316921846 (238h28m44.188004901s) [noise=true , address=145.79.227.10:4403 , piece_id=J2VDRKJLGYJ5PYJLMHE5V42YEGCZVXBSGBS43JR4IUFVURLQ63ZQ, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317050902 (238h28m44.188133937s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 15 13:57:44.317069087 (238h28m44.188152132s) [noise=true , address=145.79.227.26:4655 , piece_id=ODPVJNSDUPLZYZWFQHKA5ZTCR5HNUFNSQWHSPSKB2MCZ7OVBSW3Q, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317407521 (238h28m44.188490566s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: context canceled; context canceled Dec 15 13:57:44.317417851 (238h28m44.188500897s) [noise=true , address=145.79.227.30:4761 , piece_id=OYBTPXGKHYFYOWI56O6J2JETXC6KWMUY6F62IVR4UAOC3BG2TWMA, seg_pos={0 0}, upload=209086]: Long tail cancellation due to optimal threshold Dec 15 13:57:44.317442407 (238h28m44.188525453s) [seg_pos={0 0}, upload=209086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:57:44.317542859 (238h28m44.188625904s) [upload=209086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:57:44.317560463 (238h28m44.188643508s) [upload=209086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:57:44.761912425 (238h28m44.63299546s) [upload=209086]: Done uploading segments... Dec 15 13:57:44.761932923 (238h28m44.633015968s) [upload=209086]: Upload finished. err: <nil> Duration: 30.002766825s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.03221987s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:58:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.032249446s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX...XXX] duration: 5.001409114s trace: https://jaeger.dev.storj.io/trace/1bf9d5a38f3e5c46 FAIL (2025-12-15 13:57:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001433261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 19.343275214s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:57:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.343294801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...................X.............X..] duration: 18.214753377s trace: https://jaeger.dev.storj.io/trace/2abd5facb5a9559 FAIL (2025-12-15 13:57:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:55:17.763251994 (238h26m17.634335039s) [upload=209050]: Got next segment. Inline: false Dec 15 13:55:17.763287060 (238h26m17.634370095s) [upload=209050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:55:17.763291829 (238h26m17.634374874s) [upload=209050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:55:18.010656799 (238h26m17.881739844s) [seg_pos={0 0}, upload=209050]: Begin upload segment... Dec 15 13:55:18.010735779 (238h26m17.881818824s) [seg_pos={0 0}, upload=209050]: Done begin upload segment. Dec 15 13:55:18.010801865 (238h26m17.8818849s) [noise=true , address=77.90.31.14:4663 , piece_id=EOISLZZEL35OWEJ7X76IAICOBXLI3P62SB6LHTFI6XCXFF3OUJVQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.010854576 (238h26m17.881937621s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.011416376 (238h26m17.882499421s) [noise=true , address=145.79.227.46:4603 , piece_id=IOSJCUAGPS2E66DYV3LBLI2EMHAGIDIQRLP6M5PAEL23WSZXMQCA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.012436588 (238h26m17.883519633s) [noise=true , address=145.79.227.52:4447 , piece_id=WYIESLUUWK3A2UESTH3OSZGQO4XF55EHYAGQ4TO4MDTKVW2I6RKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.012934417 (238h26m17.884017462s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.013646021 (238h26m17.884729066s) [noise=true , address=145.79.227.29:4942 , piece_id=X6EH2SS73A4CCNGX4K67H37MGRTCFOIR4W6ZWA4MOZI7JQDCSISA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.014203713 (238h26m17.885286768s) [noise=true , address=145.79.227.32:4835 , piece_id=OKEPIKUPWAYZZECUM5RGQWZMKN46PNNIKX5RAEU5KHBK6VFZNW7Q, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.015800353 (238h26m17.886883399s) [noise=true , address=45.91.116.5:4299 , piece_id=3E5JTWYL67YFNGREU2547JWKAV4E265TQ3AR7T45IHQE7FPO7AKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.018192208 (238h26m17.889275253s) [noise=true , address=145.79.227.24:4582 , piece_id=CO6577HEQXDGYBGZ7L4P2MY6BLVVN7LFR26PEKOY5QEWGN5NGIKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.022346799 (238h26m17.893429844s) [noise=true , address=45.91.116.9:4261 , piece_id=AFE2LJJ2KGYM27HJ43TWJ2C3A4Y6FV2KUSVY4G44JGDRFV2D6UXA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.026773809 (238h26m17.897856854s) [noise=true , address=145.79.227.47:4265 , piece_id=MVSCTRULVLYN36W76AFANDSTVMNN36LQFOGSLWJKA64YE7ITHJ5A, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.028805919 (238h26m17.899888964s) [noise=true , address=145.79.227.31:4798 , piece_id=7OAVY224RKATUWAQHY47C7SJT4KY2Q4TI6UFOAA424ZHLAJN5PKQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.031835998 (238h26m17.902919044s) [noise=true , address=145.79.227.23:4546 , piece_id=SDHLZFZWQPUWJZL6R3LU3SWLYPB4JDSVOP34CXIU4H3EOEINXLDQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.032395754 (238h26m17.903478799s) [noise=true , address=77.90.31.10:4519 , piece_id=DTL3ZOAP6VLIWGOI7QWJ56XWTWUUKEMOKSRW7TDVD2PQO77DM5HA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.035231794 (238h26m17.906314839s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.036547890 (238h26m17.907630935s) [noise=true , address=145.79.227.13:4804 , piece_id=I2HWP57RPXJPTDKOC2EDYJTGIXDR5SGBRIFDAWPUWLBSVWBTWAWQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:19.926244863 (238h26m19.797327908s) [upload=209050]: Next returned nil segment Dec 15 13:55:19.926265291 (238h26m19.797348346s) [upload=209050]: Waiting for error group managing segments... Dec 15 13:55:20.224139492 (238h26m20.095222537s) [noise=true , address=77.90.31.10:4519 , piece_id=DTL3ZOAP6VLIWGOI7QWJ56XWTWUUKEMOKSRW7TDVD2PQO77DM5HA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.282598258 (238h26m20.153681303s) [noise=true , address=77.90.31.14:4663 , piece_id=EOISLZZEL35OWEJ7X76IAICOBXLI3P62SB6LHTFI6XCXFF3OUJVQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.838619711 (238h26m20.709702746s) [noise=true , address=45.91.116.9:4261 , piece_id=AFE2LJJ2KGYM27HJ43TWJ2C3A4Y6FV2KUSVY4G44JGDRFV2D6UXA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.967810176 (238h26m20.838893221s) [noise=true , address=145.79.227.24:4582 , piece_id=CO6577HEQXDGYBGZ7L4P2MY6BLVVN7LFR26PEKOY5QEWGN5NGIKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:21.069518007 (238h26m20.940601053s) [noise=true , address=145.79.227.29:4942 , piece_id=X6EH2SS73A4CCNGX4K67H37MGRTCFOIR4W6ZWA4MOZI7JQDCSISA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:21.075346817 (238h26m20.946429862s) [noise=true , address=145.79.227.47:4265 , piece_id=MVSCTRULVLYN36W76AFANDSTVMNN36LQFOGSLWJKA64YE7ITHJ5A, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:24.997273706 (238h26m24.868356751s) [noise=true , address=45.91.116.5:4299 , piece_id=3E5JTWYL67YFNGREU2547JWKAV4E265TQ3AR7T45IHQE7FPO7AKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:25.566113185 (238h26m25.43719623s) [noise=true , address=145.79.227.52:4447 , piece_id=WYIESLUUWK3A2UESTH3OSZGQO4XF55EHYAGQ4TO4MDTKVW2I6RKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:26.722871398 (238h26m26.593954433s) [noise=true , address=145.79.227.23:4546 , piece_id=SDHLZFZWQPUWJZL6R3LU3SWLYPB4JDSVOP34CXIU4H3EOEINXLDQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:26.901226052 (238h26m26.772309087s) [noise=true , address=145.79.227.32:4835 , piece_id=OKEPIKUPWAYZZECUM5RGQWZMKN46PNNIKX5RAEU5KHBK6VFZNW7Q, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:27.773855469 (238h26m27.644938524s) [noise=true , address=145.79.227.46:4603 , piece_id=IOSJCUAGPS2E66DYV3LBLI2EMHAGIDIQRLP6M5PAEL23WSZXMQCA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:32.398078231 (238h26m32.269161277s) [noise=true , address=145.79.227.31:4798 , piece_id=7OAVY224RKATUWAQHY47C7SJT4KY2Q4TI6UFOAA424ZHLAJN5PKQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:34.810695978 (238h26m34.681779013s) [noise=true , address=145.79.227.13:4804 , piece_id=I2HWP57RPXJPTDKOC2EDYJTGIXDR5SGBRIFDAWPUWLBSVWBTWAWQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:34.810715565 (238h26m34.6817986s) [seg_pos={0 0}, upload=209050]: Segment reached optimal threshold of 13 pieces. Dec 15 13:55:34.810908262 (238h26m34.681991308s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection Dec 15 13:55:34.810917440 (238h26m34.682000475s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811002282 (238h26m34.682085357s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection Dec 15 13:55:34.811014234 (238h26m34.682097269s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811221870 (238h26m34.682304905s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 15 13:55:34.811229113 (238h26m34.682312158s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811274009 (238h26m34.682357044s) [seg_pos={0 0}, upload=209050]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:55:34.811360924 (238h26m34.682443959s) [upload=209050]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:55:34.811368479 (238h26m34.682451524s) [upload=209050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:55:35.102343813 (238h26m34.973426848s) [upload=209050]: Done uploading segments... Dec 15 13:55:35.102400440 (238h26m34.973483485s) [upload=209050]: Upload finished. err: <nil> Duration: 18.214784647s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.218009009s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:57:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.21803564s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX..X..X.........................XX.......................] duration: 19.422298069s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:57:10 UTC) orbiter warning: context deadline exceeded Duration: 19.422330871s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX...XX] duration: 5.001229593s trace: https://jaeger.dev.storj.io/trace/1f42843f99d54e8 FAIL (2025-12-15 13:56:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001247797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.759336951s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:56:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.759462921s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X...................X.............X.] duration: 23.937621439s trace: https://jaeger.dev.storj.io/trace/2abd5facb5a9559 FAIL (2025-12-15 13:56:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:55:17.763251994 (238h26m17.634335039s) [upload=209050]: Got next segment. Inline: false Dec 15 13:55:17.763287060 (238h26m17.634370095s) [upload=209050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:55:17.763291829 (238h26m17.634374874s) [upload=209050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:55:18.010656799 (238h26m17.881739844s) [seg_pos={0 0}, upload=209050]: Begin upload segment... Dec 15 13:55:18.010735779 (238h26m17.881818824s) [seg_pos={0 0}, upload=209050]: Done begin upload segment. Dec 15 13:55:18.010801865 (238h26m17.8818849s) [noise=true , address=77.90.31.14:4663 , piece_id=EOISLZZEL35OWEJ7X76IAICOBXLI3P62SB6LHTFI6XCXFF3OUJVQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.010854576 (238h26m17.881937621s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.011416376 (238h26m17.882499421s) [noise=true , address=145.79.227.46:4603 , piece_id=IOSJCUAGPS2E66DYV3LBLI2EMHAGIDIQRLP6M5PAEL23WSZXMQCA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.012436588 (238h26m17.883519633s) [noise=true , address=145.79.227.52:4447 , piece_id=WYIESLUUWK3A2UESTH3OSZGQO4XF55EHYAGQ4TO4MDTKVW2I6RKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.012934417 (238h26m17.884017462s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.013646021 (238h26m17.884729066s) [noise=true , address=145.79.227.29:4942 , piece_id=X6EH2SS73A4CCNGX4K67H37MGRTCFOIR4W6ZWA4MOZI7JQDCSISA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.014203713 (238h26m17.885286768s) [noise=true , address=145.79.227.32:4835 , piece_id=OKEPIKUPWAYZZECUM5RGQWZMKN46PNNIKX5RAEU5KHBK6VFZNW7Q, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.015800353 (238h26m17.886883399s) [noise=true , address=45.91.116.5:4299 , piece_id=3E5JTWYL67YFNGREU2547JWKAV4E265TQ3AR7T45IHQE7FPO7AKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.018192208 (238h26m17.889275253s) [noise=true , address=145.79.227.24:4582 , piece_id=CO6577HEQXDGYBGZ7L4P2MY6BLVVN7LFR26PEKOY5QEWGN5NGIKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.022346799 (238h26m17.893429844s) [noise=true , address=45.91.116.9:4261 , piece_id=AFE2LJJ2KGYM27HJ43TWJ2C3A4Y6FV2KUSVY4G44JGDRFV2D6UXA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.026773809 (238h26m17.897856854s) [noise=true , address=145.79.227.47:4265 , piece_id=MVSCTRULVLYN36W76AFANDSTVMNN36LQFOGSLWJKA64YE7ITHJ5A, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.028805919 (238h26m17.899888964s) [noise=true , address=145.79.227.31:4798 , piece_id=7OAVY224RKATUWAQHY47C7SJT4KY2Q4TI6UFOAA424ZHLAJN5PKQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.031835998 (238h26m17.902919044s) [noise=true , address=145.79.227.23:4546 , piece_id=SDHLZFZWQPUWJZL6R3LU3SWLYPB4JDSVOP34CXIU4H3EOEINXLDQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.032395754 (238h26m17.903478799s) [noise=true , address=77.90.31.10:4519 , piece_id=DTL3ZOAP6VLIWGOI7QWJ56XWTWUUKEMOKSRW7TDVD2PQO77DM5HA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.035231794 (238h26m17.906314839s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.036547890 (238h26m17.907630935s) [noise=true , address=145.79.227.13:4804 , piece_id=I2HWP57RPXJPTDKOC2EDYJTGIXDR5SGBRIFDAWPUWLBSVWBTWAWQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:19.926244863 (238h26m19.797327908s) [upload=209050]: Next returned nil segment Dec 15 13:55:19.926265291 (238h26m19.797348346s) [upload=209050]: Waiting for error group managing segments... Dec 15 13:55:20.224139492 (238h26m20.095222537s) [noise=true , address=77.90.31.10:4519 , piece_id=DTL3ZOAP6VLIWGOI7QWJ56XWTWUUKEMOKSRW7TDVD2PQO77DM5HA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.282598258 (238h26m20.153681303s) [noise=true , address=77.90.31.14:4663 , piece_id=EOISLZZEL35OWEJ7X76IAICOBXLI3P62SB6LHTFI6XCXFF3OUJVQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.838619711 (238h26m20.709702746s) [noise=true , address=45.91.116.9:4261 , piece_id=AFE2LJJ2KGYM27HJ43TWJ2C3A4Y6FV2KUSVY4G44JGDRFV2D6UXA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.967810176 (238h26m20.838893221s) [noise=true , address=145.79.227.24:4582 , piece_id=CO6577HEQXDGYBGZ7L4P2MY6BLVVN7LFR26PEKOY5QEWGN5NGIKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:21.069518007 (238h26m20.940601053s) [noise=true , address=145.79.227.29:4942 , piece_id=X6EH2SS73A4CCNGX4K67H37MGRTCFOIR4W6ZWA4MOZI7JQDCSISA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:21.075346817 (238h26m20.946429862s) [noise=true , address=145.79.227.47:4265 , piece_id=MVSCTRULVLYN36W76AFANDSTVMNN36LQFOGSLWJKA64YE7ITHJ5A, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:24.997273706 (238h26m24.868356751s) [noise=true , address=45.91.116.5:4299 , piece_id=3E5JTWYL67YFNGREU2547JWKAV4E265TQ3AR7T45IHQE7FPO7AKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:25.566113185 (238h26m25.43719623s) [noise=true , address=145.79.227.52:4447 , piece_id=WYIESLUUWK3A2UESTH3OSZGQO4XF55EHYAGQ4TO4MDTKVW2I6RKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:26.722871398 (238h26m26.593954433s) [noise=true , address=145.79.227.23:4546 , piece_id=SDHLZFZWQPUWJZL6R3LU3SWLYPB4JDSVOP34CXIU4H3EOEINXLDQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:26.901226052 (238h26m26.772309087s) [noise=true , address=145.79.227.32:4835 , piece_id=OKEPIKUPWAYZZECUM5RGQWZMKN46PNNIKX5RAEU5KHBK6VFZNW7Q, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:27.773855469 (238h26m27.644938524s) [noise=true , address=145.79.227.46:4603 , piece_id=IOSJCUAGPS2E66DYV3LBLI2EMHAGIDIQRLP6M5PAEL23WSZXMQCA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:32.398078231 (238h26m32.269161277s) [noise=true , address=145.79.227.31:4798 , piece_id=7OAVY224RKATUWAQHY47C7SJT4KY2Q4TI6UFOAA424ZHLAJN5PKQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:34.810695978 (238h26m34.681779013s) [noise=true , address=145.79.227.13:4804 , piece_id=I2HWP57RPXJPTDKOC2EDYJTGIXDR5SGBRIFDAWPUWLBSVWBTWAWQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:34.810715565 (238h26m34.6817986s) [seg_pos={0 0}, upload=209050]: Segment reached optimal threshold of 13 pieces. Dec 15 13:55:34.810908262 (238h26m34.681991308s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection Dec 15 13:55:34.810917440 (238h26m34.682000475s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811002282 (238h26m34.682085357s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection Dec 15 13:55:34.811014234 (238h26m34.682097269s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811221870 (238h26m34.682304905s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 15 13:55:34.811229113 (238h26m34.682312158s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811274009 (238h26m34.682357044s) [seg_pos={0 0}, upload=209050]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:55:34.811360924 (238h26m34.682443959s) [upload=209050]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:55:34.811368479 (238h26m34.682451524s) [upload=209050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:55:35.102343813 (238h26m34.973426848s) [upload=209050]: Done uploading segments... Dec 15 13:55:35.102400440 (238h26m34.973483485s) [upload=209050]: Upload finished. err: <nil> Duration: 23.937651777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 17.585778552s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:56:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.58582431s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX...X] duration: 5.001338669s trace: https://jaeger.dev.storj.io/trace/3cd2cec8723557fb FAIL (2025-12-15 13:56:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001365401s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.385988695s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:56:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.386019694s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..XX..X..X.........................XX......................] duration: 13.95470271s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:55:51 UTC) orbiter warning: context deadline exceeded Duration: 13.95471859s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...................X.............X] duration: 30.004637212s trace: https://jaeger.dev.storj.io/trace/2abd5facb5a9559 FAIL (2025-12-15 13:55:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:55:17.763251994 (238h26m17.634335039s) [upload=209050]: Got next segment. Inline: false Dec 15 13:55:17.763287060 (238h26m17.634370095s) [upload=209050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:55:17.763291829 (238h26m17.634374874s) [upload=209050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:55:18.010656799 (238h26m17.881739844s) [seg_pos={0 0}, upload=209050]: Begin upload segment... Dec 15 13:55:18.010735779 (238h26m17.881818824s) [seg_pos={0 0}, upload=209050]: Done begin upload segment. Dec 15 13:55:18.010801865 (238h26m17.8818849s) [noise=true , address=77.90.31.14:4663 , piece_id=EOISLZZEL35OWEJ7X76IAICOBXLI3P62SB6LHTFI6XCXFF3OUJVQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.010854576 (238h26m17.881937621s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.011416376 (238h26m17.882499421s) [noise=true , address=145.79.227.46:4603 , piece_id=IOSJCUAGPS2E66DYV3LBLI2EMHAGIDIQRLP6M5PAEL23WSZXMQCA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.012436588 (238h26m17.883519633s) [noise=true , address=145.79.227.52:4447 , piece_id=WYIESLUUWK3A2UESTH3OSZGQO4XF55EHYAGQ4TO4MDTKVW2I6RKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.012934417 (238h26m17.884017462s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.013646021 (238h26m17.884729066s) [noise=true , address=145.79.227.29:4942 , piece_id=X6EH2SS73A4CCNGX4K67H37MGRTCFOIR4W6ZWA4MOZI7JQDCSISA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.014203713 (238h26m17.885286768s) [noise=true , address=145.79.227.32:4835 , piece_id=OKEPIKUPWAYZZECUM5RGQWZMKN46PNNIKX5RAEU5KHBK6VFZNW7Q, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.015800353 (238h26m17.886883399s) [noise=true , address=45.91.116.5:4299 , piece_id=3E5JTWYL67YFNGREU2547JWKAV4E265TQ3AR7T45IHQE7FPO7AKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.018192208 (238h26m17.889275253s) [noise=true , address=145.79.227.24:4582 , piece_id=CO6577HEQXDGYBGZ7L4P2MY6BLVVN7LFR26PEKOY5QEWGN5NGIKA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.022346799 (238h26m17.893429844s) [noise=true , address=45.91.116.9:4261 , piece_id=AFE2LJJ2KGYM27HJ43TWJ2C3A4Y6FV2KUSVY4G44JGDRFV2D6UXA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.026773809 (238h26m17.897856854s) [noise=true , address=145.79.227.47:4265 , piece_id=MVSCTRULVLYN36W76AFANDSTVMNN36LQFOGSLWJKA64YE7ITHJ5A, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.028805919 (238h26m17.899888964s) [noise=true , address=145.79.227.31:4798 , piece_id=7OAVY224RKATUWAQHY47C7SJT4KY2Q4TI6UFOAA424ZHLAJN5PKQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.031835998 (238h26m17.902919044s) [noise=true , address=145.79.227.23:4546 , piece_id=SDHLZFZWQPUWJZL6R3LU3SWLYPB4JDSVOP34CXIU4H3EOEINXLDQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.032395754 (238h26m17.903478799s) [noise=true , address=77.90.31.10:4519 , piece_id=DTL3ZOAP6VLIWGOI7QWJ56XWTWUUKEMOKSRW7TDVD2PQO77DM5HA, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.035231794 (238h26m17.906314839s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:18.036547890 (238h26m17.907630935s) [noise=true , address=145.79.227.13:4804 , piece_id=I2HWP57RPXJPTDKOC2EDYJTGIXDR5SGBRIFDAWPUWLBSVWBTWAWQ, seg_pos={0 0}, upload=209050]: Uploading piece... Dec 15 13:55:19.926244863 (238h26m19.797327908s) [upload=209050]: Next returned nil segment Dec 15 13:55:19.926265291 (238h26m19.797348346s) [upload=209050]: Waiting for error group managing segments... Dec 15 13:55:20.224139492 (238h26m20.095222537s) [noise=true , address=77.90.31.10:4519 , piece_id=DTL3ZOAP6VLIWGOI7QWJ56XWTWUUKEMOKSRW7TDVD2PQO77DM5HA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.282598258 (238h26m20.153681303s) [noise=true , address=77.90.31.14:4663 , piece_id=EOISLZZEL35OWEJ7X76IAICOBXLI3P62SB6LHTFI6XCXFF3OUJVQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.838619711 (238h26m20.709702746s) [noise=true , address=45.91.116.9:4261 , piece_id=AFE2LJJ2KGYM27HJ43TWJ2C3A4Y6FV2KUSVY4G44JGDRFV2D6UXA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:20.967810176 (238h26m20.838893221s) [noise=true , address=145.79.227.24:4582 , piece_id=CO6577HEQXDGYBGZ7L4P2MY6BLVVN7LFR26PEKOY5QEWGN5NGIKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:21.069518007 (238h26m20.940601053s) [noise=true , address=145.79.227.29:4942 , piece_id=X6EH2SS73A4CCNGX4K67H37MGRTCFOIR4W6ZWA4MOZI7JQDCSISA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:21.075346817 (238h26m20.946429862s) [noise=true , address=145.79.227.47:4265 , piece_id=MVSCTRULVLYN36W76AFANDSTVMNN36LQFOGSLWJKA64YE7ITHJ5A, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:24.997273706 (238h26m24.868356751s) [noise=true , address=45.91.116.5:4299 , piece_id=3E5JTWYL67YFNGREU2547JWKAV4E265TQ3AR7T45IHQE7FPO7AKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:25.566113185 (238h26m25.43719623s) [noise=true , address=145.79.227.52:4447 , piece_id=WYIESLUUWK3A2UESTH3OSZGQO4XF55EHYAGQ4TO4MDTKVW2I6RKA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:26.722871398 (238h26m26.593954433s) [noise=true , address=145.79.227.23:4546 , piece_id=SDHLZFZWQPUWJZL6R3LU3SWLYPB4JDSVOP34CXIU4H3EOEINXLDQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:26.901226052 (238h26m26.772309087s) [noise=true , address=145.79.227.32:4835 , piece_id=OKEPIKUPWAYZZECUM5RGQWZMKN46PNNIKX5RAEU5KHBK6VFZNW7Q, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:27.773855469 (238h26m27.644938524s) [noise=true , address=145.79.227.46:4603 , piece_id=IOSJCUAGPS2E66DYV3LBLI2EMHAGIDIQRLP6M5PAEL23WSZXMQCA, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:32.398078231 (238h26m32.269161277s) [noise=true , address=145.79.227.31:4798 , piece_id=7OAVY224RKATUWAQHY47C7SJT4KY2Q4TI6UFOAA424ZHLAJN5PKQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:34.810695978 (238h26m34.681779013s) [noise=true , address=145.79.227.13:4804 , piece_id=I2HWP57RPXJPTDKOC2EDYJTGIXDR5SGBRIFDAWPUWLBSVWBTWAWQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: <nil> Dec 15 13:55:34.810715565 (238h26m34.6817986s) [seg_pos={0 0}, upload=209050]: Segment reached optimal threshold of 13 pieces. Dec 15 13:55:34.810908262 (238h26m34.681991308s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58286->145.79.227.26:4653: use of closed network connection Dec 15 13:55:34.810917440 (238h26m34.682000475s) [noise=true , address=145.79.227.26:4653 , piece_id=BXKFQ2OQG3NZP5KX4GPJPCB6XWFIWED2YLQCWQCDOCF7YGAGDZTQ, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811002282 (238h26m34.682085357s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48566->145.79.227.11:4730: use of closed network connection Dec 15 13:55:34.811014234 (238h26m34.682097269s) [noise=true , address=145.79.227.11:4730 , piece_id=74RGXOEY5XW2X7CQUBHMLEFW7Q4T6ODKEBS2QBGOEK377UHSSY2Q, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811221870 (238h26m34.682304905s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 15 13:55:34.811229113 (238h26m34.682312158s) [noise=true , address=145.79.227.49:4338 , piece_id=ESIWYQW46PZBVYP6OIWYA3EER4M7NIWRT4JLTXXQSHSHCEUSFOTQ, seg_pos={0 0}, upload=209050]: Long tail cancellation due to optimal threshold Dec 15 13:55:34.811274009 (238h26m34.682357044s) [seg_pos={0 0}, upload=209050]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:55:34.811360924 (238h26m34.682443959s) [upload=209050]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:55:34.811368479 (238h26m34.682451524s) [upload=209050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:55:35.102343813 (238h26m34.973426848s) [upload=209050]: Done uploading segments... Dec 15 13:55:35.102400440 (238h26m34.973483485s) [upload=209050]: Upload finished. err: <nil> Duration: 30.004663402s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..XX..X..X.........................XX.....................] duration: 20.330435481s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:55:36 UTC) orbiter warning: context deadline exceeded Duration: 20.330464716s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX...] duration: 5.001369939s trace: https://jaeger.dev.storj.io/trace/109bf17a7c792364 FAIL (2025-12-15 13:55:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001459369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.016933824s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:55:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.016972378s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 17.602975046s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:55:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.603004142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..XX..X..X.........................XX....................] duration: 23.723269175s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:54:55 UTC) orbiter warning: context deadline exceeded Duration: 23.723298331s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.........................X...................X.............] duration: 9.31345598s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:54:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.313485235s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX..] duration: 3.893341018s trace: https://jaeger.dev.storj.io/trace/109bf17a7c792364 FAIL (2025-12-15 13:54:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.893369131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..XX..X..X.........................XX...................] duration: 28.408261116s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:54:13 UTC) orbiter warning: context deadline exceeded Duration: 28.408277917s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX.] duration: 4.652268874s trace: https://jaeger.dev.storj.io/trace/109bf17a7c792364 FAIL (2025-12-15 13:54:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.652296977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.........................X...................X............] duration: 9.382996268s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:54:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.383030783s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.905659403s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:53:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.905779502s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.729915151s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:53:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.729950358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..XX..X..X.........................XX..................] duration: 28.447944264s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:53:45 UTC) orbiter warning: context deadline exceeded Duration: 28.447971757s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 15.207411527s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:53:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.207439119s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXXX] duration: 5.00209278s trace: https://jaeger.dev.storj.io/trace/109bf17a7c792364 FAIL (2025-12-15 13:53:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00234966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 23.884328494s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:53:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.884360143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..XX..X..X.........................XX.................] duration: 20.581527338s trace: https://jaeger.dev.storj.io/trace/24b02f8b8965bdf3 FAIL (2025-12-15 13:53:10 UTC) orbiter warning: context deadline exceeded Duration: 20.581592692s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.........................X...................X...........] duration: 23.032062986s trace: https://jaeger.dev.storj.io/trace/38dcbad0c1200661 FAIL (2025-12-15 13:53:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 15 13:43:36.755362953 (238h14m36.626445989s) [upload=208889]: Starting upload Dec 15 13:43:36.755513700 (238h14m36.626596745s) [upload=208889]: Done starting upload Dec 15 13:43:36.755975249 (238h14m36.627058284s) [upload=208889]: Uploading segments... Dec 15 13:43:36.756235755 (238h14m36.6273188s) [upload=208889]: Got next segment. Inline: false Dec 15 13:43:36.756318062 (238h14m36.627401107s) [upload=208889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:43:36.756325926 (238h14m36.627408972s) [upload=208889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:43:37.707588074 (238h14m37.578671119s) [seg_pos={0 0}, upload=208889]: Begin upload segment... Dec 15 13:43:37.707663969 (238h14m37.578747005s) [seg_pos={0 0}, upload=208889]: Done begin upload segment. Dec 15 13:43:37.707710147 (238h14m37.578793192s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.707772787 (238h14m37.578855832s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708363020 (238h14m37.579446056s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708706354 (238h14m37.579789399s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.708828707 (238h14m37.579911742s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709040490 (238h14m37.580123535s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709412779 (238h14m37.580495824s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709422938 (238h14m37.580505973s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.709993906 (238h14m37.581076951s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710128251 (238h14m37.581211296s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710550616 (238h14m37.581633661s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710568870 (238h14m37.581651916s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.710993078 (238h14m37.582076123s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711006144 (238h14m37.582089189s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.711735903 (238h14m37.582818948s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:37.713658143 (238h14m37.584741278s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Uploading piece... Dec 15 13:43:39.830463293 (238h14m39.701546338s) [upload=208889]: Next returned nil segment Dec 15 13:43:39.830496466 (238h14m39.701579501s) [upload=208889]: Waiting for error group managing segments... Dec 15 13:43:40.170213382 (238h14m40.041296428s) [noise=true , address=77.90.31.14:4663 , piece_id=BEDUFT6CYD4ZO4N2VHPPL7Z4N5TFRWONJKDSDFWQJIV2BCNEA65Q, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.919447591 (238h14m40.790530636s) [noise=true , address=145.79.227.25:4617 , piece_id=B43A5K6OEZNMCW2U7O4XUUW2DA6MNG3EIRKW4QHNDMM6RV5OMKNQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.927829942 (238h14m40.798912987s) [noise=true , address=145.79.227.33:4871 , piece_id=ZCTXOIYYJJYZ6KXTXOLZWLK7REXOPKSYCK6SCCXUHPAQIPO22HMA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.936444876 (238h14m40.807527911s) [noise=true , address=145.79.227.42:4121 , piece_id=PV2LPTQ2Q3N654DBWKTWY27NY5BWOS6TFVG5QGF472ONJGZ3SNEQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.937630104 (238h14m40.808713139s) [noise=true , address=145.79.227.11:4730 , piece_id=5HSV7EGPELSBNOPXT73GQK22XW7PZVM2WZY2QJYCX5PEHDLXXRUA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.938474321 (238h14m40.809557366s) [noise=true , address=145.79.227.47:4265 , piece_id=E2TYPNADU2VI4M42P5P52OJXYGWMLKSQAM2VCC6AZPAP3AOPNTGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.941026851 (238h14m40.812109886s) [noise=true , address=145.79.227.48:4303 , piece_id=52JGDGNLF43RH3XGCMU2K6W6KY4Q5ESGJURW4EVA3DQN4O2VUMYA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.948089099 (238h14m40.819172144s) [noise=true , address=145.79.227.43:4157 , piece_id=QJAU5ZWIHFFPPTFQFP5P227UZRWCRXCLGVCAURJHC7P3CWJGE7EQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:40.959024722 (238h14m40.830107767s) [noise=true , address=145.79.227.50:4373 , piece_id=6GCWJQPRZZIL7HY4UFRN4K3YSDVBZEBAAISLX6SIUMR5ZGOG7SQA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:43:56.856297593 (238h14m56.727380639s) [noise=true , address=145.79.227.37:4565 , piece_id=U7YJWIX7OOBZZJCHMJ7JMM6R2YJA24CGYNC4JAA3G7ROK4V743HQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:03.463876429 (238h15m3.334959474s) [noise=true , address=145.79.227.41:4086 , piece_id=6DVPFUGC5YHZH4LCMTF4AM6FT3LRYWPXS4FPEYCPZYHUOI2SWKNA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: <nil> Dec 15 13:44:05.481623412 (238h15m5.352706477s) [noise=true , address=145.79.227.52:4447 , piece_id=IOLBQ3BLO75IMGP7TIYHNFNPHG25OHA6U74JM2PNF4BSHVKCVFIA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481646546 (238h15m5.352729591s) [noise=true , address=145.79.227.38:4978 , piece_id=YXJXDUHDCVH3CLNB72OH5SQI4O2E2H2AB7FNHT7MTGXG73GAIPPQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43664->145.79.227.38:4978: use of closed network connection Dec 15 13:44:05.481861325 (238h15m5.35294437s) [noise=true , address=145.79.227.16:4911 , piece_id=XMZPZMJPLSSXLKVKPMRMS5SCXJUEVW6RSAZ5JNQAKTFBXEMPMKKQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.481914706 (238h15m5.352997751s) [noise=true , address=145.79.227.28:4725 , piece_id=OMKRKINSW7OMHHMZPMOT7XL4F6ZSQHKYF5D4IRXI3XLJTXKUZIGA, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482023504 (238h15m5.353106549s) [noise=true , address=145.79.227.34:4907 , piece_id=SYXQ2FSPSAFWWHZPFPAQ4LW55OWTJNUSPG5MNK5OGUR6XXAOA3BQ, seg_pos={0 0}, upload=208889]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: expected piece hash; context deadline exceeded; EOF Dec 15 13:44:05.482052659 (238h15m5.353135694s) [seg_pos={0 0}, upload=208889]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 15 13:44:05.931274846 (238h15m5.802357901s) [upload=208889]: Done uploading segments... Dec 15 13:44:05.931322276 (238h15m5.802405321s) [upload=208889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 23.032091569s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 11.451208134s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:53:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.45124309s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 38.864440897s trace: https://jaeger.dev.storj.io/trace/56320c011e66f7de FAIL (2025-12-15 13:52:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.864457939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 7.073419412s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:52:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.073451582s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.340143954s trace: https://jaeger.dev.storj.io/trace/48a4d000bfb7a82c FAIL (2025-12-15 13:52:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.340187827s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXXX.X.XXXX.XXXXXXX.XXXXXX.XXXXXXXXXXXXXXXXX.....X.XXXXXX] duration: 5.002019488s trace: https://jaeger.dev.storj.io/trace/33c735f15f3a8045 FAIL (2025-12-15 13:52:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002087377s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)