Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......] duration: 5.001142776s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:26:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001169888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX...] duration: 5.901345531s trace: https://jaeger.dev.storj.io/trace/357cbbc5cb7faab3 FAIL (2025-12-28 08:25:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:23:09.175617819 (544h54m9.046700864s) [upload=477845]: Starting upload Dec 28 08:23:09.175689727 (544h54m9.046772772s) [upload=477845]: Done starting upload Dec 28 08:23:09.175905588 (544h54m9.046988633s) [upload=477845]: Uploading segments... Dec 28 08:23:09.175947427 (544h54m9.047030472s) [upload=477845]: Got next segment. Inline: false Dec 28 08:23:09.175977064 (544h54m9.047060109s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:23:09.175982093 (544h54m9.047065128s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:23:09.421982243 (544h54m9.293065288s) [seg_pos={0 0}, upload=477845]: Begin upload segment... Dec 28 08:23:09.422026808 (544h54m9.293109843s) [seg_pos={0 0}, upload=477845]: Done begin upload segment. Dec 28 08:23:09.422057115 (544h54m9.29314015s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422094957 (544h54m9.293178002s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422564041 (544h54m9.293647086s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423047462 (544h54m9.294130507s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423421022 (544h54m9.294504067s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423830903 (544h54m9.294913948s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424271482 (544h54m9.295354517s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424764982 (544h54m9.295848017s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426193432 (544h54m9.297276477s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426319743 (544h54m9.297402788s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.427550195 (544h54m9.29863324s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.428876791 (544h54m9.299959826s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.429307351 (544h54m9.300390386s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430654967 (544h54m9.301738002s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430911185 (544h54m9.30199423s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.432544916 (544h54m9.303627961s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:11.146567677 (544h54m11.017650722s) [upload=477845]: Next returned nil segment Dec 28 08:23:11.146583276 (544h54m11.017666321s) [upload=477845]: Waiting for error group managing segments... Dec 28 08:23:11.422054977 (544h54m11.293138012s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.433332632 (544h54m11.304415667s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.434811097 (544h54m11.305894133s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436366178 (544h54m11.307449223s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436735201 (544h54m11.307818246s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.439364938 (544h54m11.310447983s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.440764233 (544h54m11.311847268s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.447481244 (544h54m11.318564279s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.449183224 (544h54m11.320266269s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.456319463 (544h54m11.327402498s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.469118144 (544h54m11.34020118s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.668497871 (544h54m11.539580906s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689115431 (544h54m11.560198477s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689140229 (544h54m11.560223274s) [seg_pos={0 0}, upload=477845]: Segment reached optimal threshold of 13 pieces. Dec 28 08:23:11.689377189 (544h54m11.560460234s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689390656 (544h54m11.560473691s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689612297 (544h54m11.560695332s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689629240 (544h54m11.560712275s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689717888 (544h54m11.560800943s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689727326 (544h54m11.560810371s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689750591 (544h54m11.560833626s) [seg_pos={0 0}, upload=477845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:23:11.689912980 (544h54m11.560996025s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:23:11.689920815 (544h54m11.56100386s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:23:11.986828494 (544h54m11.857911539s) [upload=477845]: Done uploading segments... Dec 28 08:23:11.986851097 (544h54m11.857934142s) [upload=477845]: Upload finished. err: <nil> Duration: 5.901367504s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 8.771441202s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:25:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.771469827s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.....XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX......] duration: 4.162897699s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:25:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.162926774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.627079943s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:25:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.627113768s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX..] duration: 5.892181038s trace: https://jaeger.dev.storj.io/trace/357cbbc5cb7faab3 FAIL (2025-12-28 08:24:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:23:09.175617819 (544h54m9.046700864s) [upload=477845]: Starting upload Dec 28 08:23:09.175689727 (544h54m9.046772772s) [upload=477845]: Done starting upload Dec 28 08:23:09.175905588 (544h54m9.046988633s) [upload=477845]: Uploading segments... Dec 28 08:23:09.175947427 (544h54m9.047030472s) [upload=477845]: Got next segment. Inline: false Dec 28 08:23:09.175977064 (544h54m9.047060109s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:23:09.175982093 (544h54m9.047065128s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:23:09.421982243 (544h54m9.293065288s) [seg_pos={0 0}, upload=477845]: Begin upload segment... Dec 28 08:23:09.422026808 (544h54m9.293109843s) [seg_pos={0 0}, upload=477845]: Done begin upload segment. Dec 28 08:23:09.422057115 (544h54m9.29314015s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422094957 (544h54m9.293178002s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422564041 (544h54m9.293647086s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423047462 (544h54m9.294130507s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423421022 (544h54m9.294504067s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423830903 (544h54m9.294913948s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424271482 (544h54m9.295354517s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424764982 (544h54m9.295848017s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426193432 (544h54m9.297276477s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426319743 (544h54m9.297402788s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.427550195 (544h54m9.29863324s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.428876791 (544h54m9.299959826s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.429307351 (544h54m9.300390386s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430654967 (544h54m9.301738002s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430911185 (544h54m9.30199423s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.432544916 (544h54m9.303627961s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:11.146567677 (544h54m11.017650722s) [upload=477845]: Next returned nil segment Dec 28 08:23:11.146583276 (544h54m11.017666321s) [upload=477845]: Waiting for error group managing segments... Dec 28 08:23:11.422054977 (544h54m11.293138012s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.433332632 (544h54m11.304415667s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.434811097 (544h54m11.305894133s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436366178 (544h54m11.307449223s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436735201 (544h54m11.307818246s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.439364938 (544h54m11.310447983s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.440764233 (544h54m11.311847268s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.447481244 (544h54m11.318564279s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.449183224 (544h54m11.320266269s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.456319463 (544h54m11.327402498s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.469118144 (544h54m11.34020118s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.668497871 (544h54m11.539580906s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689115431 (544h54m11.560198477s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689140229 (544h54m11.560223274s) [seg_pos={0 0}, upload=477845]: Segment reached optimal threshold of 13 pieces. Dec 28 08:23:11.689377189 (544h54m11.560460234s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689390656 (544h54m11.560473691s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689612297 (544h54m11.560695332s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689629240 (544h54m11.560712275s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689717888 (544h54m11.560800943s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689727326 (544h54m11.560810371s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689750591 (544h54m11.560833626s) [seg_pos={0 0}, upload=477845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:23:11.689912980 (544h54m11.560996025s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:23:11.689920815 (544h54m11.56100386s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:23:11.986828494 (544h54m11.857911539s) [upload=477845]: Done uploading segments... Dec 28 08:23:11.986851097 (544h54m11.857934142s) [upload=477845]: Upload finished. err: <nil> Duration: 5.892209111s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 11.879914938s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:24:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.87993623s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX.] duration: 5.913179746s trace: https://jaeger.dev.storj.io/trace/357cbbc5cb7faab3 FAIL (2025-12-28 08:24:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:23:09.175617819 (544h54m9.046700864s) [upload=477845]: Starting upload Dec 28 08:23:09.175689727 (544h54m9.046772772s) [upload=477845]: Done starting upload Dec 28 08:23:09.175905588 (544h54m9.046988633s) [upload=477845]: Uploading segments... Dec 28 08:23:09.175947427 (544h54m9.047030472s) [upload=477845]: Got next segment. Inline: false Dec 28 08:23:09.175977064 (544h54m9.047060109s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:23:09.175982093 (544h54m9.047065128s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:23:09.421982243 (544h54m9.293065288s) [seg_pos={0 0}, upload=477845]: Begin upload segment... Dec 28 08:23:09.422026808 (544h54m9.293109843s) [seg_pos={0 0}, upload=477845]: Done begin upload segment. Dec 28 08:23:09.422057115 (544h54m9.29314015s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422094957 (544h54m9.293178002s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422564041 (544h54m9.293647086s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423047462 (544h54m9.294130507s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423421022 (544h54m9.294504067s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423830903 (544h54m9.294913948s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424271482 (544h54m9.295354517s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424764982 (544h54m9.295848017s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426193432 (544h54m9.297276477s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426319743 (544h54m9.297402788s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.427550195 (544h54m9.29863324s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.428876791 (544h54m9.299959826s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.429307351 (544h54m9.300390386s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430654967 (544h54m9.301738002s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430911185 (544h54m9.30199423s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.432544916 (544h54m9.303627961s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:11.146567677 (544h54m11.017650722s) [upload=477845]: Next returned nil segment Dec 28 08:23:11.146583276 (544h54m11.017666321s) [upload=477845]: Waiting for error group managing segments... Dec 28 08:23:11.422054977 (544h54m11.293138012s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.433332632 (544h54m11.304415667s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.434811097 (544h54m11.305894133s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436366178 (544h54m11.307449223s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436735201 (544h54m11.307818246s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.439364938 (544h54m11.310447983s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.440764233 (544h54m11.311847268s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.447481244 (544h54m11.318564279s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.449183224 (544h54m11.320266269s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.456319463 (544h54m11.327402498s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.469118144 (544h54m11.34020118s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.668497871 (544h54m11.539580906s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689115431 (544h54m11.560198477s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689140229 (544h54m11.560223274s) [seg_pos={0 0}, upload=477845]: Segment reached optimal threshold of 13 pieces. Dec 28 08:23:11.689377189 (544h54m11.560460234s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689390656 (544h54m11.560473691s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689612297 (544h54m11.560695332s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689629240 (544h54m11.560712275s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689717888 (544h54m11.560800943s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689727326 (544h54m11.560810371s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689750591 (544h54m11.560833626s) [seg_pos={0 0}, upload=477845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:23:11.689912980 (544h54m11.560996025s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:23:11.689920815 (544h54m11.56100386s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:23:11.986828494 (544h54m11.857911539s) [upload=477845]: Done uploading segments... Dec 28 08:23:11.986851097 (544h54m11.857934142s) [upload=477845]: Upload finished. err: <nil> Duration: 5.913207129s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [......XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX.....] duration: 5.000724419s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:24:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00074043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 16.042342102s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:24:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 16.042358242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX] duration: 30.002803107s trace: https://jaeger.dev.storj.io/trace/357cbbc5cb7faab3 FAIL (2025-12-28 08:23:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:23:09.175617819 (544h54m9.046700864s) [upload=477845]: Starting upload Dec 28 08:23:09.175689727 (544h54m9.046772772s) [upload=477845]: Done starting upload Dec 28 08:23:09.175905588 (544h54m9.046988633s) [upload=477845]: Uploading segments... Dec 28 08:23:09.175947427 (544h54m9.047030472s) [upload=477845]: Got next segment. Inline: false Dec 28 08:23:09.175977064 (544h54m9.047060109s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:23:09.175982093 (544h54m9.047065128s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:23:09.421982243 (544h54m9.293065288s) [seg_pos={0 0}, upload=477845]: Begin upload segment... Dec 28 08:23:09.422026808 (544h54m9.293109843s) [seg_pos={0 0}, upload=477845]: Done begin upload segment. Dec 28 08:23:09.422057115 (544h54m9.29314015s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422094957 (544h54m9.293178002s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.422564041 (544h54m9.293647086s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423047462 (544h54m9.294130507s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423421022 (544h54m9.294504067s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.423830903 (544h54m9.294913948s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424271482 (544h54m9.295354517s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.424764982 (544h54m9.295848017s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426193432 (544h54m9.297276477s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.426319743 (544h54m9.297402788s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.427550195 (544h54m9.29863324s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.428876791 (544h54m9.299959826s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.429307351 (544h54m9.300390386s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430654967 (544h54m9.301738002s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.430911185 (544h54m9.30199423s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:09.432544916 (544h54m9.303627961s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Uploading piece... Dec 28 08:23:11.146567677 (544h54m11.017650722s) [upload=477845]: Next returned nil segment Dec 28 08:23:11.146583276 (544h54m11.017666321s) [upload=477845]: Waiting for error group managing segments... Dec 28 08:23:11.422054977 (544h54m11.293138012s) [noise=true , address=145.79.227.48:4302 , piece_id=CY7577YFROU4Y22OGHZXBEVSCLTDGQELYL4I2PKJJAY4FMYZWZ4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.433332632 (544h54m11.304415667s) [noise=true , address=145.79.227.6:4258 , piece_id=WEGNJWLNOLL4TAVDTGNILYYJAAVSE3MTHZI2KLIT26A5VP5CFTVQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.434811097 (544h54m11.305894133s) [noise=true , address=145.79.227.39:4013 , piece_id=SU4RM2OJD63CN2JDUDRE6VGCD5CY7OHN4GD3WVKJCMASOZEFWLOA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436366178 (544h54m11.307449223s) [noise=true , address=145.79.227.36:4519 , piece_id=Y5S5FMN67W5IIX45XSKTBGF7SP4ARRXOTTF277UGLOSWVRZQ4I2Q, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.436735201 (544h54m11.307818246s) [noise=true , address=145.79.227.25:4619 , piece_id=GL6KHFGTI4R7ZB3BHQ3DP47DPGYBAZOOXYSAX5KKTCX3FWDAEVVA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.439364938 (544h54m11.310447983s) [noise=true , address=145.79.227.52:4447 , piece_id=R42XNOYZRQ3VR75J37C6AXK6SMEVUVOCEEL7DUNEJYGG2K7XJZ6A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.440764233 (544h54m11.311847268s) [noise=true , address=145.79.227.19:4079 , piece_id=N6HMGHSNQABIHIK3RHVAPE6TMYJFEOPGKFQWHBNYDWXDBL3YLYAA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.447481244 (544h54m11.318564279s) [noise=true , address=145.79.227.9:4365 , piece_id=Z67ZP3RFL7EELGFRS6477XQK63MTTHW35LIOAANUQMHZRTHKO5SQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.449183224 (544h54m11.320266269s) [noise=true , address=145.79.227.27:4689 , piece_id=3OVTSIJHGRAGCRNKDGTGO73LUQEGGCHKEEWRALBGOBMXDMYBV2TQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.456319463 (544h54m11.327402498s) [noise=true , address=145.79.227.14:4837 , piece_id=6IH5SDV27OQPA6CH2F4PKOP6CNBA7YBSERGYIYUKR2EYDHSZKNLA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.469118144 (544h54m11.34020118s) [noise=true , address=145.79.227.47:4266 , piece_id=UIISYIALCJJPPNDRLSLZWJLIICKZSZ6OFMM2TM47TVVASM25CU4A, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.668497871 (544h54m11.539580906s) [noise=true , address=77.90.31.10:4517 , piece_id=OZCVF4PS7THOFCH657V35Q6TAC2INS5RBG6D5TZJSGLD6M4HWXNQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689115431 (544h54m11.560198477s) [noise=true , address=45.91.116.9:4261 , piece_id=4R5GVWXUW4WTSAUQIOROISS7QO4EJNSQ2PMENNKESDDET4OKILSA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: <nil> Dec 28 08:23:11.689140229 (544h54m11.560223274s) [seg_pos={0 0}, upload=477845]: Segment reached optimal threshold of 13 pieces. Dec 28 08:23:11.689377189 (544h54m11.560460234s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689390656 (544h54m11.560473691s) [noise=true , address=45.91.116.6:4976 , piece_id=WYTQ5NIIVOLEKWOYEHTDGBBAYMAULKT4T42IPM4RRVHDQU2FOVCA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689612297 (544h54m11.560695332s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689629240 (544h54m11.560712275s) [noise=true , address=77.90.31.11:4555 , piece_id=VKJNSV4465CLYB4RMP76KC4O4XWA67OEUKMGWIBY5EEF7YHEARTQ, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689717888 (544h54m11.560800943s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 08:23:11.689727326 (544h54m11.560810371s) [noise=true , address=45.91.116.4:4937 , piece_id=2CDMDZNLD4SCPSPTA56DBAAOAYXGFMK3NMAMXNKLYWSO3GRU3TGA, seg_pos={0 0}, upload=477845]: Long tail cancellation due to optimal threshold Dec 28 08:23:11.689750591 (544h54m11.560833626s) [seg_pos={0 0}, upload=477845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:23:11.689912980 (544h54m11.560996025s) [upload=477845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:23:11.689920815 (544h54m11.56100386s) [upload=477845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:23:11.986828494 (544h54m11.857911539s) [upload=477845]: Done uploading segments... Dec 28 08:23:11.986851097 (544h54m11.857934142s) [upload=477845]: Upload finished. err: <nil> Duration: 30.002825731s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX....] duration: 5.000591125s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:23:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000606293s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 8.343233783s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:23:29 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.343267307s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX...] duration: 4.738032796s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:23:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.73825574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..X] duration: 30.003470198s trace: https://jaeger.dev.storj.io/trace/36678d7709dbfec8 FAIL (2025-12-28 08:22:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: SZHH4EQBAMR63YFRXIYGAXJ57ZCYHHFL7J46ZD7RBLAFYIVPHKTQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: LJW3NA2C5ZCKTY634DJRDLSUTSCBGYAH2NINEXCLRWJYEYJAOOUQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: QJYVQ6YICYVGVQYDGUM4KBRJQBCA2KEXF4GYMY5VXUIKOH6F4PYA) context deadline exceeded ======LOGS====== Dec 28 08:22:06.388536101 (544h53m6.259619146s) [upload=477831]: Starting upload Dec 28 08:22:06.388634699 (544h53m6.259717744s) [upload=477831]: Done starting upload Dec 28 08:22:06.388847814 (544h53m6.259930859s) [upload=477831]: Uploading segments... Dec 28 08:22:06.390885545 (544h53m6.26196858s) [upload=477831]: Got next segment. Inline: false Dec 28 08:22:06.390930811 (544h53m6.262013846s) [upload=477831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:22:06.390935160 (544h53m6.262018205s) [upload=477831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:22:06.634210471 (544h53m6.505293506s) [seg_pos={0 0}, upload=477831]: Begin upload segment... Dec 28 08:22:06.634261088 (544h53m6.505344123s) [seg_pos={0 0}, upload=477831]: Done begin upload segment. Dec 28 08:22:06.634299651 (544h53m6.505382686s) [noise=true , address=45.91.116.10:4083 , piece_id=N5H5CBC4CPLEW5OONTIWZWSSAONUABTUWKJAEOIHLV5DGQHRXOUA, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.634310100 (544h53m6.505393146s) [noise=true , address=45.91.116.8:4046 , piece_id=LJW3NA2C5ZCKTY634DJRDLSUTSCBGYAH2NINEXCLRWJYEYJAOOUQ, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.634453674 (544h53m6.505536709s) [noise=true , address=45.91.116.13:4153 , piece_id=SZHH4EQBAMR63YFRXIYGAXJ57ZCYHHFL7J46ZD7RBLAFYIVPHKTQ, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.634884415 (544h53m6.50596745s) [noise=true , address=145.79.227.45:4230 , piece_id=3MVUOXISU5AYGAIBUBJSVDJPZSZEUT64HQXPJWDR4JKSKBCZFX3A, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.635385849 (544h53m6.506468884s) [noise=true , address=145.79.227.10:4401 , piece_id=YZDAZ5GO6RXTCJQAY7MXLQARAMA5UMADB2BZNWCE4RXDXDBL5FVA, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.635826699 (544h53m6.506909734s) [noise=true , address=145.79.227.32:4836 , piece_id=CPA57EORVTPRZFOGXSVEYH4RBPN6BL7J4CF3UFKKY4HHRIHR3RPA, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.636383529 (544h53m6.507466574s) [noise=true , address=145.79.227.25:4619 , piece_id=YUSZYPGEASGY35OLFCA6TDUPH3JBIMJQR7CCGGNEK6WQIBY24KSQ, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.637659628 (544h53m6.508742673s) [noise=true , address=145.79.227.15:4875 , piece_id=G2PRIASDKROKKNSGWB3DMRTJSZLF2UICAWXKRSM77VZ2IWHTBPFQ, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.639342604 (544h53m6.510425649s) [noise=true , address=77.90.31.18:4805 , piece_id=XFJSYARMFHCB4U73XS3BDJCAKFMR5KWTNYYILRJNTGGKR6XGVAEQ, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.640978449 (544h53m6.512061484s) [noise=true , address=77.90.31.10:4520 , piece_id=QJYVQ6YICYVGVQYDGUM4KBRJQBCA2KEXF4GYMY5VXUIKOH6F4PYA, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.642614965 (544h53m6.513698s) [noise=true , address=145.79.227.5:4223 , piece_id=B5AGMWHZVLIFUYFTZMXHAVBXDBVLSM2JYKRP5ECWCEDFFOSYKT4Q, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.644183171 (544h53m6.515266206s) [noise=true , address=145.79.227.14:4839 , piece_id=2RPQZL7WOAKI4NGZHE4CCJMJFRRE5A5BDCW23JDADOTFOZ4MZE2A, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.645667938 (544h53m6.516750974s) [noise=true , address=45.91.116.4:4937 , piece_id=26TQ4FJDOUWPBR47XB4CJBVTKG5TBGLYW2LJTBGJJKDOXUHBPXJQ, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.647274748 (544h53m6.518357783s) [noise=true , address=145.79.227.36:4517 , piece_id=NIEEPVFUNGJNFABTTPNQIVHEQ2ISJH5YBKFLFTSOBXDOEYGD7RYA, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.648855729 (544h53m6.519938774s) [noise=true , address=77.90.31.19:4841 , piece_id=TXM3O7JJ3VXXMZSHT5UXJNDGWOPARAP5IHNMHSUPPFLX7WK243YA, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:06.651007475 (544h53m6.522090511s) [noise=true , address=145.79.227.52:4445 , piece_id=R3POJHGZ5MDUWRDN2CDEVMPPWGPMNVO3K4MDZ5XFMMC3JVHUYV6A, seg_pos={0 0}, upload=477831]: Uploading piece... Dec 28 08:22:08.358766650 (544h53m8.229849685s) [upload=477831]: Next returned nil segment Dec 28 08:22:08.358791637 (544h53m8.229874672s) [upload=477831]: Waiting for error group managing segments... Dec 28 08:22:08.646961878 (544h53m8.518044913s) [noise=true , address=145.79.227.32:4836 , piece_id=CPA57EORVTPRZFOGXSVEYH4RBPN6BL7J4CF3UFKKY4HHRIHR3RPA, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.647417666 (544h53m8.518500711s) [noise=true , address=145.79.227.10:4401 , piece_id=YZDAZ5GO6RXTCJQAY7MXLQARAMA5UMADB2BZNWCE4RXDXDBL5FVA, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.649947984 (544h53m8.521031029s) [noise=true , address=145.79.227.25:4619 , piece_id=YUSZYPGEASGY35OLFCA6TDUPH3JBIMJQR7CCGGNEK6WQIBY24KSQ, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.655241494 (544h53m8.526324539s) [noise=true , address=145.79.227.15:4875 , piece_id=G2PRIASDKROKKNSGWB3DMRTJSZLF2UICAWXKRSM77VZ2IWHTBPFQ, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.657052943 (544h53m8.528135978s) [noise=true , address=145.79.227.45:4230 , piece_id=3MVUOXISU5AYGAIBUBJSVDJPZSZEUT64HQXPJWDR4JKSKBCZFX3A, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.667676943 (544h53m8.538759988s) [noise=true , address=145.79.227.52:4445 , piece_id=R3POJHGZ5MDUWRDN2CDEVMPPWGPMNVO3K4MDZ5XFMMC3JVHUYV6A, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.671066798 (544h53m8.542149833s) [noise=true , address=145.79.227.5:4223 , piece_id=B5AGMWHZVLIFUYFTZMXHAVBXDBVLSM2JYKRP5ECWCEDFFOSYKT4Q, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.676922999 (544h53m8.548006044s) [noise=true , address=145.79.227.36:4517 , piece_id=NIEEPVFUNGJNFABTTPNQIVHEQ2ISJH5YBKFLFTSOBXDOEYGD7RYA, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.681075927 (544h53m8.552158972s) [noise=true , address=145.79.227.14:4839 , piece_id=2RPQZL7WOAKI4NGZHE4CCJMJFRRE5A5BDCW23JDADOTFOZ4MZE2A, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.846730605 (544h53m8.71781364s) [noise=true , address=45.91.116.13:4153 , piece_id=SZHH4EQBAMR63YFRXIYGAXJ57ZCYHHFL7J46ZD7RBLAFYIVPHKTQ, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.897434685 (544h53m8.76851782s) [noise=true , address=45.91.116.8:4046 , piece_id=LJW3NA2C5ZCKTY634DJRDLSUTSCBGYAH2NINEXCLRWJYEYJAOOUQ, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:08.899391721 (544h53m8.770474766s) [noise=true , address=77.90.31.18:4805 , piece_id=XFJSYARMFHCB4U73XS3BDJCAKFMR5KWTNYYILRJNTGGKR6XGVAEQ, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:09.045124825 (544h53m8.91620789s) [noise=true , address=77.90.31.10:4520 , piece_id=QJYVQ6YICYVGVQYDGUM4KBRJQBCA2KEXF4GYMY5VXUIKOH6F4PYA, seg_pos={0 0}, upload=477831]: Done uploading piece. err: <nil> Dec 28 08:22:09.045154020 (544h53m8.916237065s) [seg_pos={0 0}, upload=477831]: Segment reached optimal threshold of 13 pieces. Dec 28 08:22:09.045389749 (544h53m8.916472784s) [noise=true , address=45.91.116.10:4083 , piece_id=N5H5CBC4CPLEW5OONTIWZWSSAONUABTUWKJAEOIHLV5DGQHRXOUA, seg_pos={0 0}, upload=477831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 08:22:09.045398014 (544h53m8.916481059s) [noise=true , address=45.91.116.10:4083 , piece_id=N5H5CBC4CPLEW5OONTIWZWSSAONUABTUWKJAEOIHLV5DGQHRXOUA, seg_pos={0 0}, upload=477831]: Long tail cancellation due to optimal threshold Dec 28 08:22:09.045541719 (544h53m8.916624764s) [noise=true , address=77.90.31.19:4841 , piece_id=TXM3O7JJ3VXXMZSHT5UXJNDGWOPARAP5IHNMHSUPPFLX7WK243YA, seg_pos={0 0}, upload=477831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 28 08:22:09.045550676 (544h53m8.916633711s) [noise=true , address=77.90.31.19:4841 , piece_id=TXM3O7JJ3VXXMZSHT5UXJNDGWOPARAP5IHNMHSUPPFLX7WK243YA, seg_pos={0 0}, upload=477831]: Long tail cancellation due to optimal threshold Dec 28 08:22:09.045553741 (544h53m8.916636786s) [noise=true , address=45.91.116.4:4937 , piece_id=26TQ4FJDOUWPBR47XB4CJBVTKG5TBGLYW2LJTBGJJKDOXUHBPXJQ, seg_pos={0 0}, upload=477831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 08:22:09.045575744 (544h53m8.916658789s) [noise=true , address=45.91.116.4:4937 , piece_id=26TQ4FJDOUWPBR47XB4CJBVTKG5TBGLYW2LJTBGJJKDOXUHBPXJQ, seg_pos={0 0}, upload=477831]: Long tail cancellation due to optimal threshold Dec 28 08:22:09.045605950 (544h53m8.916688995s) [seg_pos={0 0}, upload=477831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:22:09.045763220 (544h53m8.916846275s) [upload=477831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:22:09.045771706 (544h53m8.916854751s) [upload=477831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:22:09.336177002 (544h53m9.207260048s) [upload=477831]: Done uploading segments... Dec 28 08:22:09.336196680 (544h53m9.207279725s) [upload=477831]: Upload finished. err: <nil> Duration: 30.003494625s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X........XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX..] duration: 5.000853153s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:22:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000879554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 11.502847832s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:22:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.502930389s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.182859473s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:21:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.182886494s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XX........XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX.] duration: 4.263302488s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:21:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.263328969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..] duration: 8.352275293s trace: https://jaeger.dev.storj.io/trace/5a20856ef3597cbe FAIL (2025-12-28 08:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A) context deadline exceeded ======LOGS====== Dec 28 08:20:18.791070497 (544h51m18.662153543s) [upload=477805]: Starting upload Dec 28 08:20:18.791143106 (544h51m18.662226151s) [upload=477805]: Done starting upload Dec 28 08:20:18.791209672 (544h51m18.662292708s) [upload=477805]: Uploading segments... Dec 28 08:20:18.791351102 (544h51m18.662434147s) [upload=477805]: Got next segment. Inline: false Dec 28 08:20:18.791390166 (544h51m18.662473211s) [upload=477805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:20:18.791394985 (544h51m18.66247803s) [upload=477805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:20:19.035789539 (544h51m18.906872584s) [seg_pos={0 0}, upload=477805]: Begin upload segment... Dec 28 08:20:19.035865283 (544h51m18.906948328s) [seg_pos={0 0}, upload=477805]: Done begin upload segment. Dec 28 08:20:19.035928884 (544h51m18.907011929s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.035934114 (544h51m18.907017159s) [noise=true , address=45.91.116.12:4226 , piece_id=USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.036489471 (544h51m18.907572507s) [noise=true , address=145.79.227.45:4230 , piece_id=QR3FOLV6WQ7E3ZGIN5HFVYQ6DBCDOP5VFVTHCSDYMMK4Z3JFYMBQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.036785124 (544h51m18.907868159s) [noise=true , address=145.79.227.42:4121 , piece_id=GMRNNJCY2YB63KQXRTXHP6PPIBGB4V4PHVBJQU2KYH2M6I7F4DHQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037134319 (544h51m18.908217354s) [noise=true , address=145.79.227.7:4293 , piece_id=RFKSZ2KGBHFLHTIPKFMB74ZJDM53TT3B55JDMPIUURIOR2BIKS6Q, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037452264 (544h51m18.908535309s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037537377 (544h51m18.908620412s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDBNXX5ILGMV7LIH2T3BA7KR3UK2JGJWYAMJBWWPDNNQBLFU2NGQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038003584 (544h51m18.909086629s) [noise=true , address=145.79.227.15:4875 , piece_id=PT7HQ6TDO7EEIF5JQGUONI7TWAT3OT3D6JGBNZPMVED6TSGA67KA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038291203 (544h51m18.909374248s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038412934 (544h51m18.909495969s) [noise=true , address=145.79.227.13:4802 , piece_id=SBOSCVKIZ5RMIRM4XRLLSUB6QKHP4C3FLZDHL7HD3HF736LXSPCA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038800221 (544h51m18.909883256s) [noise=true , address=145.79.227.18:4007 , piece_id=VORSZ3EWQPVSQIG3M6T6BTBVADORNGCEZT6SSJDKJ4R7TNYLQO7A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039156420 (544h51m18.910239465s) [noise=true , address=145.79.227.50:4374 , piece_id=YCQ7NISMLP23KZSO4BTLQP7YUIHVYM2FBANLXSKM4BMOP3IPJ27A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039579926 (544h51m18.910662961s) [noise=true , address=145.79.227.19:4078 , piece_id=6VEEWFDNA6SAYRYMYOXW4F4OX3QADRW3C7ZRVG4WD6TCLIPQWEVQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039888975 (544h51m18.91097203s) [noise=true , address=45.91.116.14:4192 , piece_id=WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.040199146 (544h51m18.911282191s) [noise=true , address=145.79.227.51:4410 , piece_id=6KHDBCJMMGVUULDQF4MT63MK5DCZKG6EEJFY36675SMMUOKD32CA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.040491262 (544h51m18.911574307s) [noise=true , address=45.91.116.13:4154 , piece_id=UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:20.661051622 (544h51m20.532134657s) [upload=477805]: Next returned nil segment Dec 28 08:20:20.661068064 (544h51m20.532151109s) [upload=477805]: Waiting for error group managing segments... Dec 28 08:20:21.041152806 (544h51m20.912235841s) [noise=true , address=145.79.227.50:4374 , piece_id=YCQ7NISMLP23KZSO4BTLQP7YUIHVYM2FBANLXSKM4BMOP3IPJ27A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.042482719 (544h51m20.913565754s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDBNXX5ILGMV7LIH2T3BA7KR3UK2JGJWYAMJBWWPDNNQBLFU2NGQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.044385542 (544h51m20.915468587s) [noise=true , address=145.79.227.51:4410 , piece_id=6KHDBCJMMGVUULDQF4MT63MK5DCZKG6EEJFY36675SMMUOKD32CA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.046131005 (544h51m20.91721405s) [noise=true , address=145.79.227.18:4007 , piece_id=VORSZ3EWQPVSQIG3M6T6BTBVADORNGCEZT6SSJDKJ4R7TNYLQO7A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.064189472 (544h51m20.935272517s) [noise=true , address=145.79.227.15:4875 , piece_id=PT7HQ6TDO7EEIF5JQGUONI7TWAT3OT3D6JGBNZPMVED6TSGA67KA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.066413026 (544h51m20.937496071s) [noise=true , address=145.79.227.42:4121 , piece_id=GMRNNJCY2YB63KQXRTXHP6PPIBGB4V4PHVBJQU2KYH2M6I7F4DHQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.067783436 (544h51m20.938866481s) [noise=true , address=145.79.227.19:4078 , piece_id=6VEEWFDNA6SAYRYMYOXW4F4OX3QADRW3C7ZRVG4WD6TCLIPQWEVQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.070015747 (544h51m20.941098792s) [noise=true , address=145.79.227.7:4293 , piece_id=RFKSZ2KGBHFLHTIPKFMB74ZJDM53TT3B55JDMPIUURIOR2BIKS6Q, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.075017211 (544h51m20.946100256s) [noise=true , address=145.79.227.13:4802 , piece_id=SBOSCVKIZ5RMIRM4XRLLSUB6QKHP4C3FLZDHL7HD3HF736LXSPCA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.076414431 (544h51m20.947497476s) [noise=true , address=145.79.227.45:4230 , piece_id=QR3FOLV6WQ7E3ZGIN5HFVYQ6DBCDOP5VFVTHCSDYMMK4Z3JFYMBQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.319600804 (544h51m21.190683839s) [noise=true , address=45.91.116.12:4226 , piece_id=USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.329257912 (544h51m21.200340948s) [noise=true , address=45.91.116.14:4192 , piece_id=WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.349718163 (544h51m21.220801208s) [noise=true , address=45.91.116.13:4154 , piece_id=UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.349743060 (544h51m21.220826095s) [seg_pos={0 0}, upload=477805]: Segment reached optimal threshold of 13 pieces. Dec 28 08:20:21.350185161 (544h51m21.221268206s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350195742 (544h51m21.221278797s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350378149 (544h51m21.221461194s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350385884 (544h51m21.221468919s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350518497 (544h51m21.221601542s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350523827 (544h51m21.221606872s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350544336 (544h51m21.221627381s) [seg_pos={0 0}, upload=477805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:20:21.350624048 (544h51m21.221707093s) [upload=477805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:20:21.350647823 (544h51m21.221730868s) [upload=477805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:20:21.636838513 (544h51m21.507921548s) [upload=477805]: Done uploading segments... Dec 28 08:20:21.636861227 (544h51m21.507944272s) [upload=477805]: Upload finished. err: <nil> Duration: 8.35230582s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XXX........XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX] duration: 5.001000153s trace: https://jaeger.dev.storj.io/trace/4a77f7727cf3aa22 FAIL (2025-12-28 08:21:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001017668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.191362706s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:21:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.191383655s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XXXX........XX......XX........X.XX..XXX.X.XX.XX.......X.XX.X] duration: 5.004708076s trace: https://jaeger.dev.storj.io/trace/4de8387027b97307 FAIL (2025-12-28 08:21:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00473666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X.] duration: 5.711982881s trace: https://jaeger.dev.storj.io/trace/5a20856ef3597cbe FAIL (2025-12-28 08:20:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A) context deadline exceeded ======LOGS====== Dec 28 08:20:18.791070497 (544h51m18.662153543s) [upload=477805]: Starting upload Dec 28 08:20:18.791143106 (544h51m18.662226151s) [upload=477805]: Done starting upload Dec 28 08:20:18.791209672 (544h51m18.662292708s) [upload=477805]: Uploading segments... Dec 28 08:20:18.791351102 (544h51m18.662434147s) [upload=477805]: Got next segment. Inline: false Dec 28 08:20:18.791390166 (544h51m18.662473211s) [upload=477805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:20:18.791394985 (544h51m18.66247803s) [upload=477805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:20:19.035789539 (544h51m18.906872584s) [seg_pos={0 0}, upload=477805]: Begin upload segment... Dec 28 08:20:19.035865283 (544h51m18.906948328s) [seg_pos={0 0}, upload=477805]: Done begin upload segment. Dec 28 08:20:19.035928884 (544h51m18.907011929s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.035934114 (544h51m18.907017159s) [noise=true , address=45.91.116.12:4226 , piece_id=USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.036489471 (544h51m18.907572507s) [noise=true , address=145.79.227.45:4230 , piece_id=QR3FOLV6WQ7E3ZGIN5HFVYQ6DBCDOP5VFVTHCSDYMMK4Z3JFYMBQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.036785124 (544h51m18.907868159s) [noise=true , address=145.79.227.42:4121 , piece_id=GMRNNJCY2YB63KQXRTXHP6PPIBGB4V4PHVBJQU2KYH2M6I7F4DHQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037134319 (544h51m18.908217354s) [noise=true , address=145.79.227.7:4293 , piece_id=RFKSZ2KGBHFLHTIPKFMB74ZJDM53TT3B55JDMPIUURIOR2BIKS6Q, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037452264 (544h51m18.908535309s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037537377 (544h51m18.908620412s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDBNXX5ILGMV7LIH2T3BA7KR3UK2JGJWYAMJBWWPDNNQBLFU2NGQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038003584 (544h51m18.909086629s) [noise=true , address=145.79.227.15:4875 , piece_id=PT7HQ6TDO7EEIF5JQGUONI7TWAT3OT3D6JGBNZPMVED6TSGA67KA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038291203 (544h51m18.909374248s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038412934 (544h51m18.909495969s) [noise=true , address=145.79.227.13:4802 , piece_id=SBOSCVKIZ5RMIRM4XRLLSUB6QKHP4C3FLZDHL7HD3HF736LXSPCA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038800221 (544h51m18.909883256s) [noise=true , address=145.79.227.18:4007 , piece_id=VORSZ3EWQPVSQIG3M6T6BTBVADORNGCEZT6SSJDKJ4R7TNYLQO7A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039156420 (544h51m18.910239465s) [noise=true , address=145.79.227.50:4374 , piece_id=YCQ7NISMLP23KZSO4BTLQP7YUIHVYM2FBANLXSKM4BMOP3IPJ27A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039579926 (544h51m18.910662961s) [noise=true , address=145.79.227.19:4078 , piece_id=6VEEWFDNA6SAYRYMYOXW4F4OX3QADRW3C7ZRVG4WD6TCLIPQWEVQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039888975 (544h51m18.91097203s) [noise=true , address=45.91.116.14:4192 , piece_id=WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.040199146 (544h51m18.911282191s) [noise=true , address=145.79.227.51:4410 , piece_id=6KHDBCJMMGVUULDQF4MT63MK5DCZKG6EEJFY36675SMMUOKD32CA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.040491262 (544h51m18.911574307s) [noise=true , address=45.91.116.13:4154 , piece_id=UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:20.661051622 (544h51m20.532134657s) [upload=477805]: Next returned nil segment Dec 28 08:20:20.661068064 (544h51m20.532151109s) [upload=477805]: Waiting for error group managing segments... Dec 28 08:20:21.041152806 (544h51m20.912235841s) [noise=true , address=145.79.227.50:4374 , piece_id=YCQ7NISMLP23KZSO4BTLQP7YUIHVYM2FBANLXSKM4BMOP3IPJ27A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.042482719 (544h51m20.913565754s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDBNXX5ILGMV7LIH2T3BA7KR3UK2JGJWYAMJBWWPDNNQBLFU2NGQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.044385542 (544h51m20.915468587s) [noise=true , address=145.79.227.51:4410 , piece_id=6KHDBCJMMGVUULDQF4MT63MK5DCZKG6EEJFY36675SMMUOKD32CA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.046131005 (544h51m20.91721405s) [noise=true , address=145.79.227.18:4007 , piece_id=VORSZ3EWQPVSQIG3M6T6BTBVADORNGCEZT6SSJDKJ4R7TNYLQO7A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.064189472 (544h51m20.935272517s) [noise=true , address=145.79.227.15:4875 , piece_id=PT7HQ6TDO7EEIF5JQGUONI7TWAT3OT3D6JGBNZPMVED6TSGA67KA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.066413026 (544h51m20.937496071s) [noise=true , address=145.79.227.42:4121 , piece_id=GMRNNJCY2YB63KQXRTXHP6PPIBGB4V4PHVBJQU2KYH2M6I7F4DHQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.067783436 (544h51m20.938866481s) [noise=true , address=145.79.227.19:4078 , piece_id=6VEEWFDNA6SAYRYMYOXW4F4OX3QADRW3C7ZRVG4WD6TCLIPQWEVQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.070015747 (544h51m20.941098792s) [noise=true , address=145.79.227.7:4293 , piece_id=RFKSZ2KGBHFLHTIPKFMB74ZJDM53TT3B55JDMPIUURIOR2BIKS6Q, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.075017211 (544h51m20.946100256s) [noise=true , address=145.79.227.13:4802 , piece_id=SBOSCVKIZ5RMIRM4XRLLSUB6QKHP4C3FLZDHL7HD3HF736LXSPCA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.076414431 (544h51m20.947497476s) [noise=true , address=145.79.227.45:4230 , piece_id=QR3FOLV6WQ7E3ZGIN5HFVYQ6DBCDOP5VFVTHCSDYMMK4Z3JFYMBQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.319600804 (544h51m21.190683839s) [noise=true , address=45.91.116.12:4226 , piece_id=USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.329257912 (544h51m21.200340948s) [noise=true , address=45.91.116.14:4192 , piece_id=WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.349718163 (544h51m21.220801208s) [noise=true , address=45.91.116.13:4154 , piece_id=UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.349743060 (544h51m21.220826095s) [seg_pos={0 0}, upload=477805]: Segment reached optimal threshold of 13 pieces. Dec 28 08:20:21.350185161 (544h51m21.221268206s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350195742 (544h51m21.221278797s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350378149 (544h51m21.221461194s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350385884 (544h51m21.221468919s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350518497 (544h51m21.221601542s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350523827 (544h51m21.221606872s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350544336 (544h51m21.221627381s) [seg_pos={0 0}, upload=477805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:20:21.350624048 (544h51m21.221707093s) [upload=477805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:20:21.350647823 (544h51m21.221730868s) [upload=477805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:20:21.636838513 (544h51m21.507921548s) [upload=477805]: Done uploading segments... Dec 28 08:20:21.636861227 (544h51m21.507944272s) [upload=477805]: Upload finished. err: <nil> Duration: 5.712538568s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 7.371827466s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:20:49 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.371856451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X] duration: 30.003382078s trace: https://jaeger.dev.storj.io/trace/5a20856ef3597cbe FAIL (2025-12-28 08:20:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A) context deadline exceeded ======LOGS====== Dec 28 08:20:18.791070497 (544h51m18.662153543s) [upload=477805]: Starting upload Dec 28 08:20:18.791143106 (544h51m18.662226151s) [upload=477805]: Done starting upload Dec 28 08:20:18.791209672 (544h51m18.662292708s) [upload=477805]: Uploading segments... Dec 28 08:20:18.791351102 (544h51m18.662434147s) [upload=477805]: Got next segment. Inline: false Dec 28 08:20:18.791390166 (544h51m18.662473211s) [upload=477805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:20:18.791394985 (544h51m18.66247803s) [upload=477805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:20:19.035789539 (544h51m18.906872584s) [seg_pos={0 0}, upload=477805]: Begin upload segment... Dec 28 08:20:19.035865283 (544h51m18.906948328s) [seg_pos={0 0}, upload=477805]: Done begin upload segment. Dec 28 08:20:19.035928884 (544h51m18.907011929s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.035934114 (544h51m18.907017159s) [noise=true , address=45.91.116.12:4226 , piece_id=USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.036489471 (544h51m18.907572507s) [noise=true , address=145.79.227.45:4230 , piece_id=QR3FOLV6WQ7E3ZGIN5HFVYQ6DBCDOP5VFVTHCSDYMMK4Z3JFYMBQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.036785124 (544h51m18.907868159s) [noise=true , address=145.79.227.42:4121 , piece_id=GMRNNJCY2YB63KQXRTXHP6PPIBGB4V4PHVBJQU2KYH2M6I7F4DHQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037134319 (544h51m18.908217354s) [noise=true , address=145.79.227.7:4293 , piece_id=RFKSZ2KGBHFLHTIPKFMB74ZJDM53TT3B55JDMPIUURIOR2BIKS6Q, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037452264 (544h51m18.908535309s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.037537377 (544h51m18.908620412s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDBNXX5ILGMV7LIH2T3BA7KR3UK2JGJWYAMJBWWPDNNQBLFU2NGQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038003584 (544h51m18.909086629s) [noise=true , address=145.79.227.15:4875 , piece_id=PT7HQ6TDO7EEIF5JQGUONI7TWAT3OT3D6JGBNZPMVED6TSGA67KA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038291203 (544h51m18.909374248s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038412934 (544h51m18.909495969s) [noise=true , address=145.79.227.13:4802 , piece_id=SBOSCVKIZ5RMIRM4XRLLSUB6QKHP4C3FLZDHL7HD3HF736LXSPCA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.038800221 (544h51m18.909883256s) [noise=true , address=145.79.227.18:4007 , piece_id=VORSZ3EWQPVSQIG3M6T6BTBVADORNGCEZT6SSJDKJ4R7TNYLQO7A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039156420 (544h51m18.910239465s) [noise=true , address=145.79.227.50:4374 , piece_id=YCQ7NISMLP23KZSO4BTLQP7YUIHVYM2FBANLXSKM4BMOP3IPJ27A, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039579926 (544h51m18.910662961s) [noise=true , address=145.79.227.19:4078 , piece_id=6VEEWFDNA6SAYRYMYOXW4F4OX3QADRW3C7ZRVG4WD6TCLIPQWEVQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.039888975 (544h51m18.91097203s) [noise=true , address=45.91.116.14:4192 , piece_id=WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.040199146 (544h51m18.911282191s) [noise=true , address=145.79.227.51:4410 , piece_id=6KHDBCJMMGVUULDQF4MT63MK5DCZKG6EEJFY36675SMMUOKD32CA, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:19.040491262 (544h51m18.911574307s) [noise=true , address=45.91.116.13:4154 , piece_id=UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ, seg_pos={0 0}, upload=477805]: Uploading piece... Dec 28 08:20:20.661051622 (544h51m20.532134657s) [upload=477805]: Next returned nil segment Dec 28 08:20:20.661068064 (544h51m20.532151109s) [upload=477805]: Waiting for error group managing segments... Dec 28 08:20:21.041152806 (544h51m20.912235841s) [noise=true , address=145.79.227.50:4374 , piece_id=YCQ7NISMLP23KZSO4BTLQP7YUIHVYM2FBANLXSKM4BMOP3IPJ27A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.042482719 (544h51m20.913565754s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDBNXX5ILGMV7LIH2T3BA7KR3UK2JGJWYAMJBWWPDNNQBLFU2NGQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.044385542 (544h51m20.915468587s) [noise=true , address=145.79.227.51:4410 , piece_id=6KHDBCJMMGVUULDQF4MT63MK5DCZKG6EEJFY36675SMMUOKD32CA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.046131005 (544h51m20.91721405s) [noise=true , address=145.79.227.18:4007 , piece_id=VORSZ3EWQPVSQIG3M6T6BTBVADORNGCEZT6SSJDKJ4R7TNYLQO7A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.064189472 (544h51m20.935272517s) [noise=true , address=145.79.227.15:4875 , piece_id=PT7HQ6TDO7EEIF5JQGUONI7TWAT3OT3D6JGBNZPMVED6TSGA67KA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.066413026 (544h51m20.937496071s) [noise=true , address=145.79.227.42:4121 , piece_id=GMRNNJCY2YB63KQXRTXHP6PPIBGB4V4PHVBJQU2KYH2M6I7F4DHQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.067783436 (544h51m20.938866481s) [noise=true , address=145.79.227.19:4078 , piece_id=6VEEWFDNA6SAYRYMYOXW4F4OX3QADRW3C7ZRVG4WD6TCLIPQWEVQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.070015747 (544h51m20.941098792s) [noise=true , address=145.79.227.7:4293 , piece_id=RFKSZ2KGBHFLHTIPKFMB74ZJDM53TT3B55JDMPIUURIOR2BIKS6Q, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.075017211 (544h51m20.946100256s) [noise=true , address=145.79.227.13:4802 , piece_id=SBOSCVKIZ5RMIRM4XRLLSUB6QKHP4C3FLZDHL7HD3HF736LXSPCA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.076414431 (544h51m20.947497476s) [noise=true , address=145.79.227.45:4230 , piece_id=QR3FOLV6WQ7E3ZGIN5HFVYQ6DBCDOP5VFVTHCSDYMMK4Z3JFYMBQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.319600804 (544h51m21.190683839s) [noise=true , address=45.91.116.12:4226 , piece_id=USHZZMXPYYSULHV5BNHMNV7TCWDCYRFKWMT2BGEIDN6VT7ZJ6U2A, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.329257912 (544h51m21.200340948s) [noise=true , address=45.91.116.14:4192 , piece_id=WIQ4UIU7EZDMWHS4H6XDLAA4CR6C6TUGAEJ5KUBUCXQ7KJ7LJEMA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.349718163 (544h51m21.220801208s) [noise=true , address=45.91.116.13:4154 , piece_id=UAO3AGBR4KW37IZ45HFKLCZGTVLEFZLYYUBKMG7MNBAATRW7UHYQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: <nil> Dec 28 08:20:21.349743060 (544h51m21.220826095s) [seg_pos={0 0}, upload=477805]: Segment reached optimal threshold of 13 pieces. Dec 28 08:20:21.350185161 (544h51m21.221268206s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350195742 (544h51m21.221278797s) [noise=true , address=77.90.31.21:4878 , piece_id=L5YDSYMKNERKBV5OKZHNKSO7MYBM5I6DOFRC7WOEXT65X2WQFGGQ, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350378149 (544h51m21.221461194s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350385884 (544h51m21.221468919s) [noise=true , address=77.90.31.10:4518 , piece_id=2XIYFBD5UFR2VJCNKAVMCFIVV6R5WP7CTG7IHN3LYF5AZML6RNFQ, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350518497 (544h51m21.221601542s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:20:21.350523827 (544h51m21.221606872s) [noise=true , address=45.91.116.5:4297 , piece_id=JUTEDBJATYODMKDQK3TUL5NTNVB6WYBDJTQETUP3YV34OCFOPXIA, seg_pos={0 0}, upload=477805]: Long tail cancellation due to optimal threshold Dec 28 08:20:21.350544336 (544h51m21.221627381s) [seg_pos={0 0}, upload=477805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:20:21.350624048 (544h51m21.221707093s) [upload=477805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:20:21.350647823 (544h51m21.221730868s) [upload=477805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:20:21.636838513 (544h51m21.507921548s) [upload=477805]: Done uploading segments... Dec 28 08:20:21.636861227 (544h51m21.507944272s) [upload=477805]: Upload finished. err: <nil> Duration: 30.003408838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XXXXX........XX......XX........X.XX..XXX.X.XX.XX.......X.XX.] duration: 5.000754314s trace: https://jaeger.dev.storj.io/trace/6b02f34aee3a759a FAIL (2025-12-28 08:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00078279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..] duration: 5.89239719s trace: https://jaeger.dev.storj.io/trace/693a4f088e948c57 FAIL (2025-12-28 08:20:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ) context deadline exceeded ======LOGS====== Dec 28 08:18:37.668799514 (544h49m37.539882559s) [upload=477782]: Starting upload Dec 28 08:18:37.668941474 (544h49m37.540024519s) [upload=477782]: Done starting upload Dec 28 08:18:37.669007459 (544h49m37.540090504s) [upload=477782]: Uploading segments... Dec 28 08:18:37.669302802 (544h49m37.540385837s) [upload=477782]: Got next segment. Inline: false Dec 28 08:18:37.669338981 (544h49m37.540422026s) [upload=477782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:18:37.669344331 (544h49m37.540427366s) [upload=477782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:18:37.917364539 (544h49m37.788447584s) [seg_pos={0 0}, upload=477782]: Begin upload segment... Dec 28 08:18:37.917412100 (544h49m37.788495135s) [seg_pos={0 0}, upload=477782]: Done begin upload segment. Dec 28 08:18:37.917441165 (544h49m37.78852421s) [noise=true , address=145.79.227.12:4767 , piece_id=G5LKYDIA3REVB3MPDFE2TBERTKSRNPBU7IYYGJOTJ2PBI4VEIWNQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.917550804 (544h49m37.788633879s) [noise=true , address=145.79.227.41:4085 , piece_id=YQS6TYYMIHBZNPF4CIJASO6RR547BO7VJB4GYPPBERPRM3HL634Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918283188 (544h49m37.789366223s) [noise=true , address=145.79.227.39:4014 , piece_id=PO5R4S2NPPJBSEXWX7RIIYLXBZ5MIHPPJERY27N3NCWIBDQ4LNRA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918478981 (544h49m37.789562026s) [noise=true , address=145.79.227.9:4365 , piece_id=2QFGX7XXEOZ6ARRUQ5TDLY5SHLZQWVKJH5L6QTUY5PBSHC62WEJA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918637232 (544h49m37.789720267s) [noise=true , address=145.79.227.43:4158 , piece_id=TTLALQD7ML4C3CEPXIZYCRYJMQ464BJCBV5IV36FXF6ERSZPBE3Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918908870 (544h49m37.789991905s) [noise=true , address=145.79.227.48:4302 , piece_id=S4NBDHV7AEMFEDUWK2PPBIPIE3TZ6NVXCVJOXNBOL3OGUBVKJE6A, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918982039 (544h49m37.790065074s) [noise=true , address=145.79.227.5:4221 , piece_id=33KBUNG3536TQBQE7SZRSJKM4ZEW3SBKGSIQCATA7OBPZCDTNUGQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919228789 (544h49m37.790311834s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919492521 (544h49m37.790575556s) [noise=true , address=145.79.227.37:4565 , piece_id=AHE5I2VXWTP7ZSBTQE2HR2UCEGHY35K2E4FZF5MYNRNSGEYEITFQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919582963 (544h49m37.790666008s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919780119 (544h49m37.790863154s) [noise=true , address=145.79.227.45:4229 , piece_id=ISWTXRU4B7XCIDBIBTJ6TU2G2XTPP2IVX4NF3XRS2SLQN7CYKUXA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.920099998 (544h49m37.791183033s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.920527102 (544h49m37.791610147s) [noise=true , address=145.79.227.51:4409 , piece_id=TQEO6N66OYIO2RZBVPPXQW6N6PZB2GMYU4XGC6AOI3QFZOLOJE3Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.921095453 (544h49m37.792178499s) [noise=true , address=45.91.116.10:4083 , piece_id=O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.921714532 (544h49m37.792797568s) [noise=true , address=77.90.31.14:4663 , piece_id=VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.924256793 (544h49m37.795339828s) [noise=true , address=45.91.116.12:4226 , piece_id=52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:39.635875137 (544h49m39.506958172s) [upload=477782]: Next returned nil segment Dec 28 08:18:39.635892980 (544h49m39.506976015s) [upload=477782]: Waiting for error group managing segments... Dec 28 08:18:39.925664089 (544h49m39.796747124s) [noise=true , address=145.79.227.51:4409 , piece_id=TQEO6N66OYIO2RZBVPPXQW6N6PZB2GMYU4XGC6AOI3QFZOLOJE3Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.926452802 (544h49m39.797535837s) [noise=true , address=145.79.227.37:4565 , piece_id=AHE5I2VXWTP7ZSBTQE2HR2UCEGHY35K2E4FZF5MYNRNSGEYEITFQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.927636465 (544h49m39.798719511s) [noise=true , address=145.79.227.39:4014 , piece_id=PO5R4S2NPPJBSEXWX7RIIYLXBZ5MIHPPJERY27N3NCWIBDQ4LNRA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.927961614 (544h49m39.799044649s) [noise=true , address=145.79.227.41:4085 , piece_id=YQS6TYYMIHBZNPF4CIJASO6RR547BO7VJB4GYPPBERPRM3HL634Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.930734945 (544h49m39.801817991s) [noise=true , address=145.79.227.5:4221 , piece_id=33KBUNG3536TQBQE7SZRSJKM4ZEW3SBKGSIQCATA7OBPZCDTNUGQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.931331621 (544h49m39.802414667s) [noise=true , address=145.79.227.43:4158 , piece_id=TTLALQD7ML4C3CEPXIZYCRYJMQ464BJCBV5IV36FXF6ERSZPBE3Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.938288950 (544h49m39.809371995s) [noise=true , address=145.79.227.12:4767 , piece_id=G5LKYDIA3REVB3MPDFE2TBERTKSRNPBU7IYYGJOTJ2PBI4VEIWNQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.941127766 (544h49m39.812210811s) [noise=true , address=145.79.227.48:4302 , piece_id=S4NBDHV7AEMFEDUWK2PPBIPIE3TZ6NVXCVJOXNBOL3OGUBVKJE6A, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.949777977 (544h49m39.820861012s) [noise=true , address=145.79.227.9:4365 , piece_id=2QFGX7XXEOZ6ARRUQ5TDLY5SHLZQWVKJH5L6QTUY5PBSHC62WEJA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.044610636 (544h49m39.915693671s) [noise=true , address=145.79.227.45:4229 , piece_id=ISWTXRU4B7XCIDBIBTJ6TU2G2XTPP2IVX4NF3XRS2SLQN7CYKUXA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.096166799 (544h49m39.967249844s) [noise=true , address=77.90.31.14:4663 , piece_id=VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.148928958 (544h49m40.020011993s) [noise=true , address=45.91.116.10:4083 , piece_id=O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.214959913 (544h49m40.086042948s) [noise=true , address=45.91.116.12:4226 , piece_id=52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.214986424 (544h49m40.086069459s) [seg_pos={0 0}, upload=477782]: Segment reached optimal threshold of 13 pieces. Dec 28 08:18:40.215266297 (544h49m40.086349342s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215275875 (544h49m40.08635892s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215553493 (544h49m40.086636538s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215561669 (544h49m40.086644704s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215821644 (544h49m40.086904709s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215846070 (544h49m40.086929125s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215917055 (544h49m40.08700009s) [seg_pos={0 0}, upload=477782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:18:40.216030281 (544h49m40.087113316s) [upload=477782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:18:40.216037335 (544h49m40.08712038s) [upload=477782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:18:40.504037832 (544h49m40.375120877s) [upload=477782]: Done uploading segments... Dec 28 08:18:40.504063080 (544h49m40.375146115s) [upload=477782]: Upload finished. err: <nil> Duration: 5.892426396s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 12.626495937s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:19:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.626526514s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.XXXXX........XX......XX........X.XX..XXX.X.XX.XX.......X.XX] duration: 5.001419503s trace: https://jaeger.dev.storj.io/trace/6b02f34aee3a759a FAIL (2025-12-28 08:19:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001440844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.822520811s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:19:17 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.822547471s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX.] duration: 6.111462152s trace: https://jaeger.dev.storj.io/trace/693a4f088e948c57 FAIL (2025-12-28 08:19:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ) context deadline exceeded ======LOGS====== Dec 28 08:18:37.668799514 (544h49m37.539882559s) [upload=477782]: Starting upload Dec 28 08:18:37.668941474 (544h49m37.540024519s) [upload=477782]: Done starting upload Dec 28 08:18:37.669007459 (544h49m37.540090504s) [upload=477782]: Uploading segments... Dec 28 08:18:37.669302802 (544h49m37.540385837s) [upload=477782]: Got next segment. Inline: false Dec 28 08:18:37.669338981 (544h49m37.540422026s) [upload=477782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:18:37.669344331 (544h49m37.540427366s) [upload=477782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:18:37.917364539 (544h49m37.788447584s) [seg_pos={0 0}, upload=477782]: Begin upload segment... Dec 28 08:18:37.917412100 (544h49m37.788495135s) [seg_pos={0 0}, upload=477782]: Done begin upload segment. Dec 28 08:18:37.917441165 (544h49m37.78852421s) [noise=true , address=145.79.227.12:4767 , piece_id=G5LKYDIA3REVB3MPDFE2TBERTKSRNPBU7IYYGJOTJ2PBI4VEIWNQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.917550804 (544h49m37.788633879s) [noise=true , address=145.79.227.41:4085 , piece_id=YQS6TYYMIHBZNPF4CIJASO6RR547BO7VJB4GYPPBERPRM3HL634Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918283188 (544h49m37.789366223s) [noise=true , address=145.79.227.39:4014 , piece_id=PO5R4S2NPPJBSEXWX7RIIYLXBZ5MIHPPJERY27N3NCWIBDQ4LNRA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918478981 (544h49m37.789562026s) [noise=true , address=145.79.227.9:4365 , piece_id=2QFGX7XXEOZ6ARRUQ5TDLY5SHLZQWVKJH5L6QTUY5PBSHC62WEJA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918637232 (544h49m37.789720267s) [noise=true , address=145.79.227.43:4158 , piece_id=TTLALQD7ML4C3CEPXIZYCRYJMQ464BJCBV5IV36FXF6ERSZPBE3Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918908870 (544h49m37.789991905s) [noise=true , address=145.79.227.48:4302 , piece_id=S4NBDHV7AEMFEDUWK2PPBIPIE3TZ6NVXCVJOXNBOL3OGUBVKJE6A, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918982039 (544h49m37.790065074s) [noise=true , address=145.79.227.5:4221 , piece_id=33KBUNG3536TQBQE7SZRSJKM4ZEW3SBKGSIQCATA7OBPZCDTNUGQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919228789 (544h49m37.790311834s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919492521 (544h49m37.790575556s) [noise=true , address=145.79.227.37:4565 , piece_id=AHE5I2VXWTP7ZSBTQE2HR2UCEGHY35K2E4FZF5MYNRNSGEYEITFQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919582963 (544h49m37.790666008s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919780119 (544h49m37.790863154s) [noise=true , address=145.79.227.45:4229 , piece_id=ISWTXRU4B7XCIDBIBTJ6TU2G2XTPP2IVX4NF3XRS2SLQN7CYKUXA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.920099998 (544h49m37.791183033s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.920527102 (544h49m37.791610147s) [noise=true , address=145.79.227.51:4409 , piece_id=TQEO6N66OYIO2RZBVPPXQW6N6PZB2GMYU4XGC6AOI3QFZOLOJE3Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.921095453 (544h49m37.792178499s) [noise=true , address=45.91.116.10:4083 , piece_id=O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.921714532 (544h49m37.792797568s) [noise=true , address=77.90.31.14:4663 , piece_id=VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.924256793 (544h49m37.795339828s) [noise=true , address=45.91.116.12:4226 , piece_id=52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:39.635875137 (544h49m39.506958172s) [upload=477782]: Next returned nil segment Dec 28 08:18:39.635892980 (544h49m39.506976015s) [upload=477782]: Waiting for error group managing segments... Dec 28 08:18:39.925664089 (544h49m39.796747124s) [noise=true , address=145.79.227.51:4409 , piece_id=TQEO6N66OYIO2RZBVPPXQW6N6PZB2GMYU4XGC6AOI3QFZOLOJE3Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.926452802 (544h49m39.797535837s) [noise=true , address=145.79.227.37:4565 , piece_id=AHE5I2VXWTP7ZSBTQE2HR2UCEGHY35K2E4FZF5MYNRNSGEYEITFQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.927636465 (544h49m39.798719511s) [noise=true , address=145.79.227.39:4014 , piece_id=PO5R4S2NPPJBSEXWX7RIIYLXBZ5MIHPPJERY27N3NCWIBDQ4LNRA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.927961614 (544h49m39.799044649s) [noise=true , address=145.79.227.41:4085 , piece_id=YQS6TYYMIHBZNPF4CIJASO6RR547BO7VJB4GYPPBERPRM3HL634Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.930734945 (544h49m39.801817991s) [noise=true , address=145.79.227.5:4221 , piece_id=33KBUNG3536TQBQE7SZRSJKM4ZEW3SBKGSIQCATA7OBPZCDTNUGQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.931331621 (544h49m39.802414667s) [noise=true , address=145.79.227.43:4158 , piece_id=TTLALQD7ML4C3CEPXIZYCRYJMQ464BJCBV5IV36FXF6ERSZPBE3Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.938288950 (544h49m39.809371995s) [noise=true , address=145.79.227.12:4767 , piece_id=G5LKYDIA3REVB3MPDFE2TBERTKSRNPBU7IYYGJOTJ2PBI4VEIWNQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.941127766 (544h49m39.812210811s) [noise=true , address=145.79.227.48:4302 , piece_id=S4NBDHV7AEMFEDUWK2PPBIPIE3TZ6NVXCVJOXNBOL3OGUBVKJE6A, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.949777977 (544h49m39.820861012s) [noise=true , address=145.79.227.9:4365 , piece_id=2QFGX7XXEOZ6ARRUQ5TDLY5SHLZQWVKJH5L6QTUY5PBSHC62WEJA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.044610636 (544h49m39.915693671s) [noise=true , address=145.79.227.45:4229 , piece_id=ISWTXRU4B7XCIDBIBTJ6TU2G2XTPP2IVX4NF3XRS2SLQN7CYKUXA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.096166799 (544h49m39.967249844s) [noise=true , address=77.90.31.14:4663 , piece_id=VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.148928958 (544h49m40.020011993s) [noise=true , address=45.91.116.10:4083 , piece_id=O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.214959913 (544h49m40.086042948s) [noise=true , address=45.91.116.12:4226 , piece_id=52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.214986424 (544h49m40.086069459s) [seg_pos={0 0}, upload=477782]: Segment reached optimal threshold of 13 pieces. Dec 28 08:18:40.215266297 (544h49m40.086349342s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215275875 (544h49m40.08635892s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215553493 (544h49m40.086636538s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215561669 (544h49m40.086644704s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215821644 (544h49m40.086904709s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215846070 (544h49m40.086929125s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215917055 (544h49m40.08700009s) [seg_pos={0 0}, upload=477782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:18:40.216030281 (544h49m40.087113316s) [upload=477782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:18:40.216037335 (544h49m40.08712038s) [upload=477782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:18:40.504037832 (544h49m40.375120877s) [upload=477782]: Done uploading segments... Dec 28 08:18:40.504063080 (544h49m40.375146115s) [upload=477782]: Upload finished. err: <nil> Duration: 6.111479705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..XXXXX........XX......XX........X.XX..XXX.X.XX.XX.......X.X] duration: 5.001514713s trace: https://jaeger.dev.storj.io/trace/7a95c5489a9576b9 FAIL (2025-12-28 08:19:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001541555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX] duration: 30.013476182s trace: https://jaeger.dev.storj.io/trace/693a4f088e948c57 FAIL (2025-12-28 08:19:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ) context deadline exceeded ======LOGS====== Dec 28 08:18:37.668799514 (544h49m37.539882559s) [upload=477782]: Starting upload Dec 28 08:18:37.668941474 (544h49m37.540024519s) [upload=477782]: Done starting upload Dec 28 08:18:37.669007459 (544h49m37.540090504s) [upload=477782]: Uploading segments... Dec 28 08:18:37.669302802 (544h49m37.540385837s) [upload=477782]: Got next segment. Inline: false Dec 28 08:18:37.669338981 (544h49m37.540422026s) [upload=477782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:18:37.669344331 (544h49m37.540427366s) [upload=477782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:18:37.917364539 (544h49m37.788447584s) [seg_pos={0 0}, upload=477782]: Begin upload segment... Dec 28 08:18:37.917412100 (544h49m37.788495135s) [seg_pos={0 0}, upload=477782]: Done begin upload segment. Dec 28 08:18:37.917441165 (544h49m37.78852421s) [noise=true , address=145.79.227.12:4767 , piece_id=G5LKYDIA3REVB3MPDFE2TBERTKSRNPBU7IYYGJOTJ2PBI4VEIWNQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.917550804 (544h49m37.788633879s) [noise=true , address=145.79.227.41:4085 , piece_id=YQS6TYYMIHBZNPF4CIJASO6RR547BO7VJB4GYPPBERPRM3HL634Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918283188 (544h49m37.789366223s) [noise=true , address=145.79.227.39:4014 , piece_id=PO5R4S2NPPJBSEXWX7RIIYLXBZ5MIHPPJERY27N3NCWIBDQ4LNRA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918478981 (544h49m37.789562026s) [noise=true , address=145.79.227.9:4365 , piece_id=2QFGX7XXEOZ6ARRUQ5TDLY5SHLZQWVKJH5L6QTUY5PBSHC62WEJA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918637232 (544h49m37.789720267s) [noise=true , address=145.79.227.43:4158 , piece_id=TTLALQD7ML4C3CEPXIZYCRYJMQ464BJCBV5IV36FXF6ERSZPBE3Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918908870 (544h49m37.789991905s) [noise=true , address=145.79.227.48:4302 , piece_id=S4NBDHV7AEMFEDUWK2PPBIPIE3TZ6NVXCVJOXNBOL3OGUBVKJE6A, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.918982039 (544h49m37.790065074s) [noise=true , address=145.79.227.5:4221 , piece_id=33KBUNG3536TQBQE7SZRSJKM4ZEW3SBKGSIQCATA7OBPZCDTNUGQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919228789 (544h49m37.790311834s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919492521 (544h49m37.790575556s) [noise=true , address=145.79.227.37:4565 , piece_id=AHE5I2VXWTP7ZSBTQE2HR2UCEGHY35K2E4FZF5MYNRNSGEYEITFQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919582963 (544h49m37.790666008s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.919780119 (544h49m37.790863154s) [noise=true , address=145.79.227.45:4229 , piece_id=ISWTXRU4B7XCIDBIBTJ6TU2G2XTPP2IVX4NF3XRS2SLQN7CYKUXA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.920099998 (544h49m37.791183033s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.920527102 (544h49m37.791610147s) [noise=true , address=145.79.227.51:4409 , piece_id=TQEO6N66OYIO2RZBVPPXQW6N6PZB2GMYU4XGC6AOI3QFZOLOJE3Q, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.921095453 (544h49m37.792178499s) [noise=true , address=45.91.116.10:4083 , piece_id=O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.921714532 (544h49m37.792797568s) [noise=true , address=77.90.31.14:4663 , piece_id=VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:37.924256793 (544h49m37.795339828s) [noise=true , address=45.91.116.12:4226 , piece_id=52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ, seg_pos={0 0}, upload=477782]: Uploading piece... Dec 28 08:18:39.635875137 (544h49m39.506958172s) [upload=477782]: Next returned nil segment Dec 28 08:18:39.635892980 (544h49m39.506976015s) [upload=477782]: Waiting for error group managing segments... Dec 28 08:18:39.925664089 (544h49m39.796747124s) [noise=true , address=145.79.227.51:4409 , piece_id=TQEO6N66OYIO2RZBVPPXQW6N6PZB2GMYU4XGC6AOI3QFZOLOJE3Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.926452802 (544h49m39.797535837s) [noise=true , address=145.79.227.37:4565 , piece_id=AHE5I2VXWTP7ZSBTQE2HR2UCEGHY35K2E4FZF5MYNRNSGEYEITFQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.927636465 (544h49m39.798719511s) [noise=true , address=145.79.227.39:4014 , piece_id=PO5R4S2NPPJBSEXWX7RIIYLXBZ5MIHPPJERY27N3NCWIBDQ4LNRA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.927961614 (544h49m39.799044649s) [noise=true , address=145.79.227.41:4085 , piece_id=YQS6TYYMIHBZNPF4CIJASO6RR547BO7VJB4GYPPBERPRM3HL634Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.930734945 (544h49m39.801817991s) [noise=true , address=145.79.227.5:4221 , piece_id=33KBUNG3536TQBQE7SZRSJKM4ZEW3SBKGSIQCATA7OBPZCDTNUGQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.931331621 (544h49m39.802414667s) [noise=true , address=145.79.227.43:4158 , piece_id=TTLALQD7ML4C3CEPXIZYCRYJMQ464BJCBV5IV36FXF6ERSZPBE3Q, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.938288950 (544h49m39.809371995s) [noise=true , address=145.79.227.12:4767 , piece_id=G5LKYDIA3REVB3MPDFE2TBERTKSRNPBU7IYYGJOTJ2PBI4VEIWNQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.941127766 (544h49m39.812210811s) [noise=true , address=145.79.227.48:4302 , piece_id=S4NBDHV7AEMFEDUWK2PPBIPIE3TZ6NVXCVJOXNBOL3OGUBVKJE6A, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:39.949777977 (544h49m39.820861012s) [noise=true , address=145.79.227.9:4365 , piece_id=2QFGX7XXEOZ6ARRUQ5TDLY5SHLZQWVKJH5L6QTUY5PBSHC62WEJA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.044610636 (544h49m39.915693671s) [noise=true , address=145.79.227.45:4229 , piece_id=ISWTXRU4B7XCIDBIBTJ6TU2G2XTPP2IVX4NF3XRS2SLQN7CYKUXA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.096166799 (544h49m39.967249844s) [noise=true , address=77.90.31.14:4663 , piece_id=VJVOUQM5YUIHJB7CWRA6GBSQ5Z755MMVTRLGGDIXZGIJRQ3SRIZQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.148928958 (544h49m40.020011993s) [noise=true , address=45.91.116.10:4083 , piece_id=O3YVLXU6ZDVBHPVM6HMWCJTIQUAKTDHA6E4AIPRPV25YYRN34UKA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.214959913 (544h49m40.086042948s) [noise=true , address=45.91.116.12:4226 , piece_id=52VGRODEXPKAD5FDOQRGHI42SCYT7OS7SFKJVCJKTH3G55HOU5KQ, seg_pos={0 0}, upload=477782]: Done uploading piece. err: <nil> Dec 28 08:18:40.214986424 (544h49m40.086069459s) [seg_pos={0 0}, upload=477782]: Segment reached optimal threshold of 13 pieces. Dec 28 08:18:40.215266297 (544h49m40.086349342s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215275875 (544h49m40.08635892s) [noise=true , address=45.91.116.5:4298 , piece_id=3OIAQWIAMXX6V4QJA7UUFPWRU5JWOXRQES7SFQQVKTDF23LNHAHA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215553493 (544h49m40.086636538s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215561669 (544h49m40.086644704s) [noise=true , address=77.90.31.10:4517 , piece_id=2B3TVPGWMXTX2VJKF7SY2YBTHASUIIOFTN42NV4ZMUUK33ZPHHYA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215821644 (544h49m40.086904709s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 08:18:40.215846070 (544h49m40.086929125s) [noise=true , address=45.91.116.8:4045 , piece_id=VZCFZ6GLGXWHELMWO2N6KOQ7BFLXJM6TTC5GBINIZDGQDISGLMHA, seg_pos={0 0}, upload=477782]: Long tail cancellation due to optimal threshold Dec 28 08:18:40.215917055 (544h49m40.08700009s) [seg_pos={0 0}, upload=477782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:18:40.216030281 (544h49m40.087113316s) [upload=477782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:18:40.216037335 (544h49m40.08712038s) [upload=477782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:18:40.504037832 (544h49m40.375120877s) [upload=477782]: Done uploading segments... Dec 28 08:18:40.504063080 (544h49m40.375146115s) [upload=477782]: Upload finished. err: <nil> Duration: 30.013497312s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 14.188943831s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:18:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 14.188967827s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...XXXXX........XX......XX........X.XX..XXX.X.XX.XX.......X.] duration: 5.001866482s trace: https://jaeger.dev.storj.io/trace/4dc095ce0f189aab FAIL (2025-12-28 08:18:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001885859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.X] duration: 30.004891071s trace: https://jaeger.dev.storj.io/trace/72a9d6fae6cd68e9 FAIL (2025-12-28 08:18:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 7JUSSOUTLDHIYWOAJZ5HLENBBVWRKL73WQVLDLPJW2Q7EF3OSFKA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: WXFMHOE7K5OKEAVH2EQG6FCDDIU3BACER7DBCPPKN2UBVGTRNWYA) context deadline exceeded ======LOGS====== 8]: Starting upload Dec 28 08:17:36.485037388 (544h48m36.356120443s) [upload=477768]: Done starting upload Dec 28 08:17:36.485107070 (544h48m36.356190126s) [upload=477768]: Uploading segments... Dec 28 08:17:36.485336117 (544h48m36.356419152s) [upload=477768]: Got next segment. Inline: false Dec 28 08:17:36.485378407 (544h48m36.356461442s) [upload=477768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:17:36.485384669 (544h48m36.356467704s) [upload=477768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:17:36.729751586 (544h48m36.600834621s) [seg_pos={0 0}, upload=477768]: Begin upload segment... Dec 28 08:17:36.729997334 (544h48m36.601080369s) [seg_pos={0 0}, upload=477768]: Done begin upload segment. Dec 28 08:17:36.730043842 (544h48m36.601126887s) [noise=true , address=145.79.227.51:4411 , piece_id=BHTTUKQIKTCDZR7ZNYIZKWV5RMW5USJBK7F2BOURV6LB5XE46PNQ, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.730379983 (544h48m36.601463018s) [noise=true , address=77.90.31.17:4770 , piece_id=3C43PIQYHGE56GPHHTGFW52TZPCPDLGNAF4HVDNAEWWILNOTCLLA, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.731015963 (544h48m36.602098998s) [noise=true , address=45.91.116.8:4045 , piece_id=WXFMHOE7K5OKEAVH2EQG6FCDDIU3BACER7DBCPPKN2UBVGTRNWYA, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.731536885 (544h48m36.60261992s) [noise=true , address=145.79.227.20:4437 , piece_id=54RZ5KA6I3GXG2TWHX4MA5DGVJ25FTVNAOBI2VCAQPAPJO526GJQ, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.732069700 (544h48m36.603152736s) [noise=true , address=145.79.227.36:4518 , piece_id=LU3Y2ETYQH5DOEOXENWSS4MKLIWQI5Z5NXB2AMI2XY6WQ67TX4JA, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.732417703 (544h48m36.603500738s) [noise=true , address=145.79.227.11:4732 , piece_id=YRCZ7L3QVOMMZCKMLGD4G3D7MPRYBBYTTYR4GRKSXXIRSI3S2GFQ, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.732736580 (544h48m36.603819615s) [noise=true , address=45.91.116.9:4262 , piece_id=7JUSSOUTLDHIYWOAJZ5HLENBBVWRKL73WQVLDLPJW2Q7EF3OSFKA, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.732844085 (544h48m36.60392712s) [noise=true , address=145.79.227.19:4077 , piece_id=IWZB2R46GOIDXZTRYQB2BB7RVDMOJU2PMH4LJVZFWRHLEZZW4VWQ, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.733030309 (544h48m36.604113354s) [noise=true , address=145.79.227.45:4230 , piece_id=KN5AAQLLG6PDI7N4CTHXFTYUZQMVCFN3EZJFHSMLK3YLCBTWRZWA, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.733335981 (544h48m36.604419026s) [noise=true , address=45.91.116.6:4973 , piece_id=L4S3LSU65TFYOBT7PMCCPC6OMJHJAYCTVX3MHOA4OXZ4GCUMGI5Q, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.733517767 (544h48m36.604600812s) [noise=true , address=77.90.31.10:4520 , piece_id=6VITQPRHJRT76YNGLH7GXMLXXP7FCFMNSY24PMY5ANCA4BEZHN3Q, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.733640281 (544h48m36.604723316s) [noise=true , address=145.79.227.9:4365 , piece_id=HO2FN6QQ6JHBPHLBJKITEHPDWLGDTETR675PDYY7226TVEB5Z7BQ, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.734381763 (544h48m36.605464798s) [noise=true , address=145.79.227.15:4876 , piece_id=HCKF253E225CWZ56SR47A6HOZ5OAAF77S5WBWQRZCXGPL56JHYRA, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.735003296 (544h48m36.606086351s) [noise=true , address=145.79.227.16:4911 , piece_id=DR5MFB32FXDGIZ7KNLA55AETVVQHNQB3R2EUM2TZLBIYEAAMGW7Q, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.736654440 (544h48m36.607737485s) [noise=true , address=145.79.227.12:4765 , piece_id=GVNBJS2SJPBIAX4AUQBALIG4WE52K4KCVFIGPNRHG2ADB7IHDRLA, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:36.738220171 (544h48m36.609303216s) [noise=true , address=145.79.227.7:4295 , piece_id=Y3DJSQWMPTE43H5F4I4SAJ4TXI6SMIVDINVGFY4YHT7LLMND5WCQ, seg_pos={0 0}, upload=477768]: Uploading piece... Dec 28 08:17:38.462549933 (544h48m38.333632979s) [upload=477768]: Next returned nil segment Dec 28 08:17:38.462562978 (544h48m38.333646023s) [upload=477768]: Waiting for error group managing segments... Dec 28 08:17:38.740379430 (544h48m38.611462465s) [noise=true , address=145.79.227.15:4876 , piece_id=HCKF253E225CWZ56SR47A6HOZ5OAAF77S5WBWQRZCXGPL56JHYRA, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.743060615 (544h48m38.61414366s) [noise=true , address=145.79.227.11:4732 , piece_id=YRCZ7L3QVOMMZCKMLGD4G3D7MPRYBBYTTYR4GRKSXXIRSI3S2GFQ, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.747129293 (544h48m38.618212328s) [noise=true , address=145.79.227.19:4077 , piece_id=IWZB2R46GOIDXZTRYQB2BB7RVDMOJU2PMH4LJVZFWRHLEZZW4VWQ, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.755753514 (544h48m38.626836549s) [noise=true , address=145.79.227.36:4518 , piece_id=LU3Y2ETYQH5DOEOXENWSS4MKLIWQI5Z5NXB2AMI2XY6WQ67TX4JA, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.756796640 (544h48m38.627879685s) [noise=true , address=145.79.227.9:4365 , piece_id=HO2FN6QQ6JHBPHLBJKITEHPDWLGDTETR675PDYY7226TVEB5Z7BQ, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.760140768 (544h48m38.631223803s) [noise=true , address=145.79.227.51:4411 , piece_id=BHTTUKQIKTCDZR7ZNYIZKWV5RMW5USJBK7F2BOURV6LB5XE46PNQ, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.762823186 (544h48m38.633906231s) [noise=true , address=145.79.227.16:4911 , piece_id=DR5MFB32FXDGIZ7KNLA55AETVVQHNQB3R2EUM2TZLBIYEAAMGW7Q, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.769777178 (544h48m38.640860213s) [noise=true , address=145.79.227.7:4295 , piece_id=Y3DJSQWMPTE43H5F4I4SAJ4TXI6SMIVDINVGFY4YHT7LLMND5WCQ, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.772808740 (544h48m38.643891785s) [noise=true , address=145.79.227.12:4765 , piece_id=GVNBJS2SJPBIAX4AUQBALIG4WE52K4KCVFIGPNRHG2ADB7IHDRLA, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.797448899 (544h48m38.668531934s) [noise=true , address=145.79.227.20:4437 , piece_id=54RZ5KA6I3GXG2TWHX4MA5DGVJ25FTVNAOBI2VCAQPAPJO526GJQ, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:38.977584158 (544h48m38.848667213s) [noise=true , address=145.79.227.45:4230 , piece_id=KN5AAQLLG6PDI7N4CTHXFTYUZQMVCFN3EZJFHSMLK3YLCBTWRZWA, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:39.058977705 (544h48m38.930060741s) [noise=true , address=45.91.116.8:4045 , piece_id=WXFMHOE7K5OKEAVH2EQG6FCDDIU3BACER7DBCPPKN2UBVGTRNWYA, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:39.284846008 (544h48m39.155929053s) [noise=true , address=45.91.116.9:4262 , piece_id=7JUSSOUTLDHIYWOAJZ5HLENBBVWRKL73WQVLDLPJW2Q7EF3OSFKA, seg_pos={0 0}, upload=477768]: Done uploading piece. err: <nil> Dec 28 08:17:39.284866607 (544h48m39.155949642s) [seg_pos={0 0}, upload=477768]: Segment reached optimal threshold of 13 pieces. Dec 28 08:17:39.285073030 (544h48m39.156156075s) [noise=true , address=77.90.31.17:4770 , piece_id=3C43PIQYHGE56GPHHTGFW52TZPCPDLGNAF4HVDNAEWWILNOTCLLA, seg_pos={0 0}, upload=477768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:44312->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:44312->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->77.90.31.17:4770: use of closed network connection Dec 28 08:17:39.285084492 (544h48m39.156167527s) [noise=true , address=77.90.31.17:4770 , piece_id=3C43PIQYHGE56GPHHTGFW52TZPCPDLGNAF4HVDNAEWWILNOTCLLA, seg_pos={0 0}, upload=477768]: Long tail cancellation due to optimal threshold Dec 28 08:17:39.285179613 (544h48m39.156262648s) [noise=true , address=77.90.31.10:4520 , piece_id=6VITQPRHJRT76YNGLH7GXMLXXP7FCFMNSY24PMY5ANCA4BEZHN3Q, seg_pos={0 0}, upload=477768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 28 08:17:39.285188209 (544h48m39.156271244s) [noise=true , address=77.90.31.10:4520 , piece_id=6VITQPRHJRT76YNGLH7GXMLXXP7FCFMNSY24PMY5ANCA4BEZHN3Q, seg_pos={0 0}, upload=477768]: Long tail cancellation due to optimal threshold Dec 28 08:17:39.285415713 (544h48m39.156498758s) [noise=true , address=45.91.116.6:4973 , piece_id=L4S3LSU65TFYOBT7PMCCPC6OMJHJAYCTVX3MHOA4OXZ4GCUMGI5Q, seg_pos={0 0}, upload=477768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:46286->45.91.116.6:4973: use of closed network connection; context canceled Dec 28 08:17:39.285422525 (544h48m39.156505561s) [noise=true , address=45.91.116.6:4973 , piece_id=L4S3LSU65TFYOBT7PMCCPC6OMJHJAYCTVX3MHOA4OXZ4GCUMGI5Q, seg_pos={0 0}, upload=477768]: Long tail cancellation due to optimal threshold Dec 28 08:17:39.285437393 (544h48m39.156520428s) [seg_pos={0 0}, upload=477768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:17:39.285517847 (544h48m39.156600892s) [upload=477768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:17:39.285523598 (544h48m39.156606643s) [upload=477768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:17:39.576477905 (544h48m39.44756094s) [upload=477768]: Done uploading segments... Dec 28 08:17:39.576499917 (544h48m39.447582952s) [upload=477768]: Upload finished. err: <nil> Duration: 30.004917931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X...XXXXX........XX......XX........X.XX..XXX.X.XX.XX.......X] duration: 5.001263013s trace: https://jaeger.dev.storj.io/trace/4dc095ce0f189aab FAIL (2025-12-28 08:17:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001288392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.541039233s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:17:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.54105384s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 13.22826165s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:17:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 13.228290514s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X...XXXXX........XX......XX........X.XX..XXX.X.XX.XX.......] duration: 4.776101784s trace: https://jaeger.dev.storj.io/trace/7e64d80e4f92f960 FAIL (2025-12-28 08:17:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.776128505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 12.789025987s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:17:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.789051045s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.......X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.] duration: 5.643537602s trace: https://jaeger.dev.storj.io/trace/f448170a36dc98b FAIL (2025-12-28 08:16:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:15:57.465108833 (544h46m57.336191878s) [upload=477748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:15:57.465115285 (544h46m57.33619835s) [upload=477748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:15:57.703471847 (544h46m57.574554892s) [seg_pos={0 0}, upload=477748]: Begin upload segment... Dec 28 08:15:57.703524207 (544h46m57.574607252s) [seg_pos={0 0}, upload=477748]: Done begin upload segment. Dec 28 08:15:57.703559494 (544h46m57.574642529s) [noise=true , address=45.91.116.9:4261 , piece_id=H3IM523NWYT4IKIKTZB3DQAXCEMDIWHNEBP7CFCS7CVPXLVKNUGA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.703588860 (544h46m57.574671895s) [noise=true , address=45.91.116.13:4155 , piece_id=57EELRGQ3DF5PYOMTWGE7LLMUUKS3ZERFL2VR5ZH3M5C44H7QBAQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.704263725 (544h46m57.57534677s) [noise=true , address=145.79.227.50:4375 , piece_id=UPTYGXGHUHVFZYEWLPQU4Y25DFFRAFU6UGY7ZIHVSUJZRRMKIXCA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.704679036 (544h46m57.575762081s) [noise=true , address=45.91.116.8:4047 , piece_id=TFGJU5MOFLANYQX4P7DED3DBNDBOIREYSIGPBIP2E2QU4P5OODFQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.705140795 (544h46m57.57622384s) [noise=true , address=145.79.227.27:4689 , piece_id=VLQ2XULJZMLY6C47HKCQ5KOKFSCR7FNB3NXSCCC7I6OHTHMPHCYQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.705628924 (544h46m57.576711969s) [noise=true , address=45.91.116.12:4225 , piece_id=PO3B4IMGNK2OW72TGHE6YSANIXTJ5INGQ2DR5TBB752766W4WD7A, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706141010 (544h46m57.577224055s) [noise=true , address=45.91.116.7:4009 , piece_id=RE3RRG3BVI6R53ZC56JK2GRDPPTJ6VG47E7YHOC3QRRQMPSCB3PQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706153283 (544h46m57.577236328s) [noise=true , address=77.90.31.21:4880 , piece_id=MR7NFMEVMNCK4LBB6XEOV6UUAQK43HJHWRPF7PTGIJYWWPYNK46Q, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706543946 (544h46m57.577626992s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6EL7ZPJSXKBXA7A5QGLWTD4TDL4YFJRTKWRNL7QXZX4OFS2E3KQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706580888 (544h46m57.577663923s) [noise=true , address=77.90.31.18:4808 , piece_id=FFEMO65VGT6XQX4KVY66BILW3AXJ32AP2FMLJ6ATOHZGCNW3KPVA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707017869 (544h46m57.578100914s) [noise=true , address=145.79.227.32:4836 , piece_id=TIYMDD56EJMBFMMMYXSXB3AH4PKQIMUABMP5P3XUQON4JFVTVJDA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707047897 (544h46m57.578130942s) [noise=true , address=45.91.116.10:4084 , piece_id=D36JQA4CVUUYCNUVP5ZBIKJHDSXTIPX4WETLQBR4WXKKQ3NVY2FA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707466113 (544h46m57.578549158s) [noise=true , address=77.90.31.12:4590 , piece_id=FSEENB5OUIZEG24JKACFQKSU644LTSRNQ27A2EWLMUKT6EGJ3B4Q, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707563889 (544h46m57.578646924s) [noise=true , address=45.91.116.6:4976 , piece_id=PO4V52BLGAIV4UDTSZEM3Q4LXHD5X5VHAUDXMFXOAVLFREX3HJCA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707946828 (544h46m57.579029864s) [noise=true , address=145.79.227.45:4229 , piece_id=CY73E7C2OCL4PWJFLL4OIQ7OH4Q54WJBBWITFPPGRAGHMAQENXSQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.708335879 (544h46m57.579418924s) [noise=true , address=145.79.227.46:4602 , piece_id=KVQQ2GTXRXBNMOETCT7B7F5VPCPC2KYCXFBWBHMDVOZZDMG2ATJA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:59.428031307 (544h46m59.299114352s) [upload=477748]: Next returned nil segment Dec 28 08:15:59.428044031 (544h46m59.299127066s) [upload=477748]: Waiting for error group managing segments... Dec 28 08:15:59.711899732 (544h46m59.582982767s) [noise=true , address=145.79.227.50:4375 , piece_id=UPTYGXGHUHVFZYEWLPQU4Y25DFFRAFU6UGY7ZIHVSUJZRRMKIXCA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.713061924 (544h46m59.584144969s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6EL7ZPJSXKBXA7A5QGLWTD4TDL4YFJRTKWRNL7QXZX4OFS2E3KQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.720189146 (544h46m59.591272181s) [noise=true , address=145.79.227.46:4602 , piece_id=KVQQ2GTXRXBNMOETCT7B7F5VPCPC2KYCXFBWBHMDVOZZDMG2ATJA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.730260133 (544h46m59.601343178s) [noise=true , address=145.79.227.27:4689 , piece_id=VLQ2XULJZMLY6C47HKCQ5KOKFSCR7FNB3NXSCCC7I6OHTHMPHCYQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.733708278 (544h46m59.604791313s) [noise=true , address=145.79.227.32:4836 , piece_id=TIYMDD56EJMBFMMMYXSXB3AH4PKQIMUABMP5P3XUQON4JFVTVJDA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.902669600 (544h46m59.773752635s) [noise=true , address=77.90.31.12:4590 , piece_id=FSEENB5OUIZEG24JKACFQKSU644LTSRNQ27A2EWLMUKT6EGJ3B4Q, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.954557273 (544h46m59.825640308s) [noise=true , address=45.91.116.6:4976 , piece_id=PO4V52BLGAIV4UDTSZEM3Q4LXHD5X5VHAUDXMFXOAVLFREX3HJCA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.973222065 (544h46m59.84430511s) [noise=true , address=77.90.31.18:4808 , piece_id=FFEMO65VGT6XQX4KVY66BILW3AXJ32AP2FMLJ6ATOHZGCNW3KPVA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.007161601 (544h46m59.878244646s) [noise=true , address=45.91.116.8:4047 , piece_id=TFGJU5MOFLANYQX4P7DED3DBNDBOIREYSIGPBIP2E2QU4P5OODFQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.033794945 (544h46m59.904878s) [noise=true , address=45.91.116.7:4009 , piece_id=RE3RRG3BVI6R53ZC56JK2GRDPPTJ6VG47E7YHOC3QRRQMPSCB3PQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.115864299 (544h46m59.986947334s) [noise=true , address=45.91.116.9:4261 , piece_id=H3IM523NWYT4IKIKTZB3DQAXCEMDIWHNEBP7CFCS7CVPXLVKNUGA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.191895175 (544h47m0.06297821s) [noise=true , address=45.91.116.10:4084 , piece_id=D36JQA4CVUUYCNUVP5ZBIKJHDSXTIPX4WETLQBR4WXKKQ3NVY2FA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.332405912 (544h47m0.203488947s) [noise=true , address=145.79.227.45:4229 , piece_id=CY73E7C2OCL4PWJFLL4OIQ7OH4Q54WJBBWITFPPGRAGHMAQENXSQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.332429817 (544h47m0.203512862s) [seg_pos={0 0}, upload=477748]: Segment reached optimal threshold of 13 pieces. Dec 28 08:16:00.332697016 (544h47m0.203780061s) [noise=true , address=45.91.116.13:4155 , piece_id=57EELRGQ3DF5PYOMTWGE7LLMUUKS3ZERFL2VR5ZH3M5C44H7QBAQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:33802->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:33802->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33802->45.91.116.13:4155: use of closed network connection Dec 28 08:16:00.332708338 (544h47m0.203791373s) [noise=true , address=45.91.116.13:4155 , piece_id=57EELRGQ3DF5PYOMTWGE7LLMUUKS3ZERFL2VR5ZH3M5C44H7QBAQ, seg_pos={0 0}, upload=477748]: Long tail cancellation due to optimal threshold Dec 28 08:16:00.332724990 (544h47m0.203808045s) [noise=true , address=77.90.31.21:4880 , piece_id=MR7NFMEVMNCK4LBB6XEOV6UUAQK43HJHWRPF7PTGIJYWWPYNK46Q, seg_pos={0 0}, upload=477748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:33414->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:33414->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33414->77.90.31.21:4880: use of closed network connection Dec 28 08:16:00.332734818 (544h47m0.203817853s) [noise=true , address=77.90.31.21:4880 , piece_id=MR7NFMEVMNCK4LBB6XEOV6UUAQK43HJHWRPF7PTGIJYWWPYNK46Q, seg_pos={0 0}, upload=477748]: Long tail cancellation due to optimal threshold Dec 28 08:16:00.332992028 (544h47m0.204075073s) [noise=true , address=45.91.116.12:4225 , piece_id=PO3B4IMGNK2OW72TGHE6YSANIXTJ5INGQ2DR5TBB752766W4WD7A, seg_pos={0 0}, upload=477748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:37234->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:37234->45.91.116.12:4225: use of closed network connection Dec 28 08:16:00.333001957 (544h47m0.204085002s) [noise=true , address=45.91.116.12:4225 , piece_id=PO3B4IMGNK2OW72TGHE6YSANIXTJ5INGQ2DR5TBB752766W4WD7A, seg_pos={0 0}, upload=477748]: Long tail cancellation due to optimal threshold Dec 28 08:16:00.333025211 (544h47m0.204108256s) [seg_pos={0 0}, upload=477748]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:16:00.333109201 (544h47m0.204192246s) [upload=477748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:16:00.333117106 (544h47m0.204200161s) [upload=477748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:16:00.614196751 (544h47m0.485279786s) [upload=477748]: Done uploading segments... Dec 28 08:16:00.614216399 (544h47m0.485299434s) [upload=477748]: Upload finished. err: <nil> Duration: 5.643561066s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..X...XXXXX........XX......XX........X.XX..XXX.X.XX.XX......] duration: 5.000940335s trace: https://jaeger.dev.storj.io/trace/7e64d80e4f92f960 FAIL (2025-12-28 08:16:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000960133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.927608415s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:16:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.927630065s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [........X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X] duration: 30.003365158s trace: https://jaeger.dev.storj.io/trace/f448170a36dc98b FAIL (2025-12-28 08:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:15:57.465108833 (544h46m57.336191878s) [upload=477748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:15:57.465115285 (544h46m57.33619835s) [upload=477748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:15:57.703471847 (544h46m57.574554892s) [seg_pos={0 0}, upload=477748]: Begin upload segment... Dec 28 08:15:57.703524207 (544h46m57.574607252s) [seg_pos={0 0}, upload=477748]: Done begin upload segment. Dec 28 08:15:57.703559494 (544h46m57.574642529s) [noise=true , address=45.91.116.9:4261 , piece_id=H3IM523NWYT4IKIKTZB3DQAXCEMDIWHNEBP7CFCS7CVPXLVKNUGA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.703588860 (544h46m57.574671895s) [noise=true , address=45.91.116.13:4155 , piece_id=57EELRGQ3DF5PYOMTWGE7LLMUUKS3ZERFL2VR5ZH3M5C44H7QBAQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.704263725 (544h46m57.57534677s) [noise=true , address=145.79.227.50:4375 , piece_id=UPTYGXGHUHVFZYEWLPQU4Y25DFFRAFU6UGY7ZIHVSUJZRRMKIXCA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.704679036 (544h46m57.575762081s) [noise=true , address=45.91.116.8:4047 , piece_id=TFGJU5MOFLANYQX4P7DED3DBNDBOIREYSIGPBIP2E2QU4P5OODFQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.705140795 (544h46m57.57622384s) [noise=true , address=145.79.227.27:4689 , piece_id=VLQ2XULJZMLY6C47HKCQ5KOKFSCR7FNB3NXSCCC7I6OHTHMPHCYQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.705628924 (544h46m57.576711969s) [noise=true , address=45.91.116.12:4225 , piece_id=PO3B4IMGNK2OW72TGHE6YSANIXTJ5INGQ2DR5TBB752766W4WD7A, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706141010 (544h46m57.577224055s) [noise=true , address=45.91.116.7:4009 , piece_id=RE3RRG3BVI6R53ZC56JK2GRDPPTJ6VG47E7YHOC3QRRQMPSCB3PQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706153283 (544h46m57.577236328s) [noise=true , address=77.90.31.21:4880 , piece_id=MR7NFMEVMNCK4LBB6XEOV6UUAQK43HJHWRPF7PTGIJYWWPYNK46Q, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706543946 (544h46m57.577626992s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6EL7ZPJSXKBXA7A5QGLWTD4TDL4YFJRTKWRNL7QXZX4OFS2E3KQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.706580888 (544h46m57.577663923s) [noise=true , address=77.90.31.18:4808 , piece_id=FFEMO65VGT6XQX4KVY66BILW3AXJ32AP2FMLJ6ATOHZGCNW3KPVA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707017869 (544h46m57.578100914s) [noise=true , address=145.79.227.32:4836 , piece_id=TIYMDD56EJMBFMMMYXSXB3AH4PKQIMUABMP5P3XUQON4JFVTVJDA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707047897 (544h46m57.578130942s) [noise=true , address=45.91.116.10:4084 , piece_id=D36JQA4CVUUYCNUVP5ZBIKJHDSXTIPX4WETLQBR4WXKKQ3NVY2FA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707466113 (544h46m57.578549158s) [noise=true , address=77.90.31.12:4590 , piece_id=FSEENB5OUIZEG24JKACFQKSU644LTSRNQ27A2EWLMUKT6EGJ3B4Q, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707563889 (544h46m57.578646924s) [noise=true , address=45.91.116.6:4976 , piece_id=PO4V52BLGAIV4UDTSZEM3Q4LXHD5X5VHAUDXMFXOAVLFREX3HJCA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.707946828 (544h46m57.579029864s) [noise=true , address=145.79.227.45:4229 , piece_id=CY73E7C2OCL4PWJFLL4OIQ7OH4Q54WJBBWITFPPGRAGHMAQENXSQ, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:57.708335879 (544h46m57.579418924s) [noise=true , address=145.79.227.46:4602 , piece_id=KVQQ2GTXRXBNMOETCT7B7F5VPCPC2KYCXFBWBHMDVOZZDMG2ATJA, seg_pos={0 0}, upload=477748]: Uploading piece... Dec 28 08:15:59.428031307 (544h46m59.299114352s) [upload=477748]: Next returned nil segment Dec 28 08:15:59.428044031 (544h46m59.299127066s) [upload=477748]: Waiting for error group managing segments... Dec 28 08:15:59.711899732 (544h46m59.582982767s) [noise=true , address=145.79.227.50:4375 , piece_id=UPTYGXGHUHVFZYEWLPQU4Y25DFFRAFU6UGY7ZIHVSUJZRRMKIXCA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.713061924 (544h46m59.584144969s) [noise=true , address=145.79.227.5:4223 , piece_id=Y6EL7ZPJSXKBXA7A5QGLWTD4TDL4YFJRTKWRNL7QXZX4OFS2E3KQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.720189146 (544h46m59.591272181s) [noise=true , address=145.79.227.46:4602 , piece_id=KVQQ2GTXRXBNMOETCT7B7F5VPCPC2KYCXFBWBHMDVOZZDMG2ATJA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.730260133 (544h46m59.601343178s) [noise=true , address=145.79.227.27:4689 , piece_id=VLQ2XULJZMLY6C47HKCQ5KOKFSCR7FNB3NXSCCC7I6OHTHMPHCYQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.733708278 (544h46m59.604791313s) [noise=true , address=145.79.227.32:4836 , piece_id=TIYMDD56EJMBFMMMYXSXB3AH4PKQIMUABMP5P3XUQON4JFVTVJDA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.902669600 (544h46m59.773752635s) [noise=true , address=77.90.31.12:4590 , piece_id=FSEENB5OUIZEG24JKACFQKSU644LTSRNQ27A2EWLMUKT6EGJ3B4Q, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.954557273 (544h46m59.825640308s) [noise=true , address=45.91.116.6:4976 , piece_id=PO4V52BLGAIV4UDTSZEM3Q4LXHD5X5VHAUDXMFXOAVLFREX3HJCA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:15:59.973222065 (544h46m59.84430511s) [noise=true , address=77.90.31.18:4808 , piece_id=FFEMO65VGT6XQX4KVY66BILW3AXJ32AP2FMLJ6ATOHZGCNW3KPVA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.007161601 (544h46m59.878244646s) [noise=true , address=45.91.116.8:4047 , piece_id=TFGJU5MOFLANYQX4P7DED3DBNDBOIREYSIGPBIP2E2QU4P5OODFQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.033794945 (544h46m59.904878s) [noise=true , address=45.91.116.7:4009 , piece_id=RE3RRG3BVI6R53ZC56JK2GRDPPTJ6VG47E7YHOC3QRRQMPSCB3PQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.115864299 (544h46m59.986947334s) [noise=true , address=45.91.116.9:4261 , piece_id=H3IM523NWYT4IKIKTZB3DQAXCEMDIWHNEBP7CFCS7CVPXLVKNUGA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.191895175 (544h47m0.06297821s) [noise=true , address=45.91.116.10:4084 , piece_id=D36JQA4CVUUYCNUVP5ZBIKJHDSXTIPX4WETLQBR4WXKKQ3NVY2FA, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.332405912 (544h47m0.203488947s) [noise=true , address=145.79.227.45:4229 , piece_id=CY73E7C2OCL4PWJFLL4OIQ7OH4Q54WJBBWITFPPGRAGHMAQENXSQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: <nil> Dec 28 08:16:00.332429817 (544h47m0.203512862s) [seg_pos={0 0}, upload=477748]: Segment reached optimal threshold of 13 pieces. Dec 28 08:16:00.332697016 (544h47m0.203780061s) [noise=true , address=45.91.116.13:4155 , piece_id=57EELRGQ3DF5PYOMTWGE7LLMUUKS3ZERFL2VR5ZH3M5C44H7QBAQ, seg_pos={0 0}, upload=477748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:33802->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:33802->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33802->45.91.116.13:4155: use of closed network connection Dec 28 08:16:00.332708338 (544h47m0.203791373s) [noise=true , address=45.91.116.13:4155 , piece_id=57EELRGQ3DF5PYOMTWGE7LLMUUKS3ZERFL2VR5ZH3M5C44H7QBAQ, seg_pos={0 0}, upload=477748]: Long tail cancellation due to optimal threshold Dec 28 08:16:00.332724990 (544h47m0.203808045s) [noise=true , address=77.90.31.21:4880 , piece_id=MR7NFMEVMNCK4LBB6XEOV6UUAQK43HJHWRPF7PTGIJYWWPYNK46Q, seg_pos={0 0}, upload=477748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:33414->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:33414->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33414->77.90.31.21:4880: use of closed network connection Dec 28 08:16:00.332734818 (544h47m0.203817853s) [noise=true , address=77.90.31.21:4880 , piece_id=MR7NFMEVMNCK4LBB6XEOV6UUAQK43HJHWRPF7PTGIJYWWPYNK46Q, seg_pos={0 0}, upload=477748]: Long tail cancellation due to optimal threshold Dec 28 08:16:00.332992028 (544h47m0.204075073s) [noise=true , address=45.91.116.12:4225 , piece_id=PO3B4IMGNK2OW72TGHE6YSANIXTJ5INGQ2DR5TBB752766W4WD7A, seg_pos={0 0}, upload=477748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:37234->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:37234->45.91.116.12:4225: use of closed network connection Dec 28 08:16:00.333001957 (544h47m0.204085002s) [noise=true , address=45.91.116.12:4225 , piece_id=PO3B4IMGNK2OW72TGHE6YSANIXTJ5INGQ2DR5TBB752766W4WD7A, seg_pos={0 0}, upload=477748]: Long tail cancellation due to optimal threshold Dec 28 08:16:00.333025211 (544h47m0.204108256s) [seg_pos={0 0}, upload=477748]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:16:00.333109201 (544h47m0.204192246s) [upload=477748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:16:00.333117106 (544h47m0.204200161s) [upload=477748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:16:00.614196751 (544h47m0.485279786s) [upload=477748]: Done uploading segments... Dec 28 08:16:00.614216399 (544h47m0.485299434s) [upload=477748]: Upload finished. err: <nil> Duration: 30.003392911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X..X...XXXXX........XX......XX........X.XX..XXX.X.XX.XX.....] duration: 3.98279928s trace: https://jaeger.dev.storj.io/trace/7e64d80e4f92f960 FAIL (2025-12-28 08:16:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.982827853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 16.510461684s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:16:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 16.510485108s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X..X...XXXXX........XX......XX........X.XX..XXX.X.XX.XX....] duration: 4.277433865s trace: https://jaeger.dev.storj.io/trace/7e64d80e4f92f960 FAIL (2025-12-28 08:15:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.277503297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X........X...X........XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.] duration: 5.864254024s trace: https://jaeger.dev.storj.io/trace/3c25324382e8e0e1 FAIL (2025-12-28 08:15:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: DCWUMBZNYRFZGXSBXCMOVEPSFO37ZYS26KWKMCZ3ZZ7VMCJJALDQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: GZDF6I25FUF2AJJHSHV3DJBQLQESPDDMSPS2YWUDZSFLFE5AR26Q) context deadline exceeded ======LOGS====== Dec 28 08:14:45.793338479 (544h45m45.664421525s) [upload=477727]: Starting upload Dec 28 08:14:45.793424063 (544h45m45.664507118s) [upload=477727]: Done starting upload Dec 28 08:14:45.793562416 (544h45m45.664645461s) [upload=477727]: Uploading segments... Dec 28 08:14:45.794162960 (544h45m45.665246005s) [upload=477727]: Got next segment. Inline: false Dec 28 08:14:45.794208857 (544h45m45.665291892s) [upload=477727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:14:45.794214888 (544h45m45.665297923s) [upload=477727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:14:46.042106387 (544h45m45.913189422s) [seg_pos={0 0}, upload=477727]: Begin upload segment... Dec 28 08:14:46.042153236 (544h45m45.913236281s) [seg_pos={0 0}, upload=477727]: Done begin upload segment. Dec 28 08:14:46.042183534 (544h45m45.913266569s) [noise=true , address=145.79.227.52:4445 , piece_id=BGQINX4VE5IRWMWIEVA2BFYGJJLDKEJGO554WRS6UVDDK5E45G2Q, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.042293122 (544h45m45.913376157s) [noise=true , address=145.79.227.9:4365 , piece_id=REWDQ7EGCFEP3IQDFRRKOZLDXFYRCWNNBN5AB7SFMT67BY5HBIFA, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.042813774 (544h45m45.913896819s) [noise=true , address=145.79.227.13:4804 , piece_id=OYC23MXF4ZEOX4XNE63NRBI4KYCJPS4Q36EUEW27WNK3EQN3PHVQ, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.043234685 (544h45m45.91431773s) [noise=true , address=145.79.227.14:4837 , piece_id=MRPF2YALSL3UTQEL3NSL3FQE7Q4YSUQBDOA3WORWQ47JRSHEXIWA, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.043634006 (544h45m45.914717051s) [noise=true , address=145.79.227.16:4909 , piece_id=HEBST442OM34XNYE5MO7EKZ4QQCYYQRQM7IIXSIUBQUBJ3JGKAFA, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.043740689 (544h45m45.914823734s) [noise=true , address=145.79.227.17:4970 , piece_id=DRHBAT2DQYF7PQ2PFTUKTE2QKWQ2BV4LHOBIPXLHKLP77P4GF37A, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.043912056 (544h45m45.914995091s) [noise=true , address=145.79.227.4:4187 , piece_id=KSXQPSPD4HRMBCNVCJBKK4GDI2KGNKICHECSK4QDQZ3GBTXX3AOQ, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.044068964 (544h45m45.915152009s) [noise=true , address=45.91.116.4:4939 , piece_id=G5XXE5S6TLE4BDXV3BDR2UVIOROKB3E5CUOUAUXQSNOBMEXTOEXA, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.044223679 (544h45m45.915306714s) [noise=true , address=45.91.116.14:4192 , piece_id=ARO7JTWXRAJ4CQ5TGIJ5JTW552MJPXVEUREJOM5CMVPECNBH33KQ, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.044329100 (544h45m45.915412145s) [noise=true , address=45.91.116.6:4975 , piece_id=TK2BG723QNKAFEZPB4DPALN3B3GFU3GOI2RSHGGQYMJ5UEDNSIIQ, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.044539180 (544h45m45.915622215s) [noise=true , address=145.79.227.11:4732 , piece_id=JS4EXZZ3I2TVBCCEQKYGWVQTRTBN6Z6SCNWP2X5BIQKL5D2W5IEA, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.044586800 (544h45m45.915669835s) [noise=true , address=77.90.31.19:4842 , piece_id=DCWUMBZNYRFZGXSBXCMOVEPSFO37ZYS26KWKMCZ3ZZ7VMCJJALDQ, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.044810457 (544h45m45.915893492s) [noise=true , address=145.79.227.5:4221 , piece_id=DOUBMPPZMT6E7NVBVTVF2RBQDL46YGUHXN4XUNW74YLYY33KOH7Q, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.044833550 (544h45m45.915916595s) [noise=true , address=45.91.116.11:4117 , piece_id=GZDF6I25FUF2AJJHSHV3DJBQLQESPDDMSPS2YWUDZSFLFE5AR26Q, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.045059170 (544h45m45.916142205s) [noise=true , address=145.79.227.41:4085 , piece_id=GBTT7QPYFNPEZYBDBYX2C5JY7MKICBDBQZGADQFXBET7HPPCWWDQ, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:46.047682294 (544h45m45.91876534s) [noise=true , address=45.91.116.13:4153 , piece_id=5QH6SO4TVK35SZECPI2V2SQGIMOYFE5ZUEZSR7LGZE5XBKMKY5YQ, seg_pos={0 0}, upload=477727]: Uploading piece... Dec 28 08:14:47.754466206 (544h45m47.625549281s) [upload=477727]: Next returned nil segment Dec 28 08:14:47.754484271 (544h45m47.625567316s) [upload=477727]: Waiting for error group managing segments... Dec 28 08:14:48.042645209 (544h45m47.913728245s) [noise=true , address=145.79.227.41:4085 , piece_id=GBTT7QPYFNPEZYBDBYX2C5JY7MKICBDBQZGADQFXBET7HPPCWWDQ, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.047631644 (544h45m47.918714689s) [noise=true , address=145.79.227.17:4970 , piece_id=DRHBAT2DQYF7PQ2PFTUKTE2QKWQ2BV4LHOBIPXLHKLP77P4GF37A, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.048669240 (544h45m47.919752285s) [noise=true , address=145.79.227.14:4837 , piece_id=MRPF2YALSL3UTQEL3NSL3FQE7Q4YSUQBDOA3WORWQ47JRSHEXIWA, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.048877086 (544h45m47.919960121s) [noise=true , address=145.79.227.16:4909 , piece_id=HEBST442OM34XNYE5MO7EKZ4QQCYYQRQM7IIXSIUBQUBJ3JGKAFA, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.053229754 (544h45m47.924312799s) [noise=true , address=145.79.227.11:4732 , piece_id=JS4EXZZ3I2TVBCCEQKYGWVQTRTBN6Z6SCNWP2X5BIQKL5D2W5IEA, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.057363506 (544h45m47.928446541s) [noise=true , address=145.79.227.5:4221 , piece_id=DOUBMPPZMT6E7NVBVTVF2RBQDL46YGUHXN4XUNW74YLYY33KOH7Q, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.058131648 (544h45m47.929214694s) [noise=true , address=145.79.227.52:4445 , piece_id=BGQINX4VE5IRWMWIEVA2BFYGJJLDKEJGO554WRS6UVDDK5E45G2Q, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.070094978 (544h45m47.941178013s) [noise=true , address=145.79.227.9:4365 , piece_id=REWDQ7EGCFEP3IQDFRRKOZLDXFYRCWNNBN5AB7SFMT67BY5HBIFA, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.075507194 (544h45m47.946590239s) [noise=true , address=145.79.227.13:4804 , piece_id=OYC23MXF4ZEOX4XNE63NRBI4KYCJPS4Q36EUEW27WNK3EQN3PHVQ, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.090320912 (544h45m47.961403957s) [noise=true , address=145.79.227.4:4187 , piece_id=KSXQPSPD4HRMBCNVCJBKK4GDI2KGNKICHECSK4QDQZ3GBTXX3AOQ, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.232281589 (544h45m48.103364624s) [noise=true , address=77.90.31.19:4842 , piece_id=DCWUMBZNYRFZGXSBXCMOVEPSFO37ZYS26KWKMCZ3ZZ7VMCJJALDQ, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.380451671 (544h45m48.251534706s) [noise=true , address=45.91.116.6:4975 , piece_id=TK2BG723QNKAFEZPB4DPALN3B3GFU3GOI2RSHGGQYMJ5UEDNSIIQ, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.396712705 (544h45m48.26779575s) [noise=true , address=45.91.116.11:4117 , piece_id=GZDF6I25FUF2AJJHSHV3DJBQLQESPDDMSPS2YWUDZSFLFE5AR26Q, seg_pos={0 0}, upload=477727]: Done uploading piece. err: <nil> Dec 28 08:14:48.396736210 (544h45m48.267819255s) [seg_pos={0 0}, upload=477727]: Segment reached optimal threshold of 13 pieces. Dec 28 08:14:48.396997307 (544h45m48.268080342s) [noise=true , address=45.91.116.13:4153 , piece_id=5QH6SO4TVK35SZECPI2V2SQGIMOYFE5ZUEZSR7LGZE5XBKMKY5YQ, seg_pos={0 0}, upload=477727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 08:14:48.397008649 (544h45m48.268091684s) [noise=true , address=45.91.116.13:4153 , piece_id=5QH6SO4TVK35SZECPI2V2SQGIMOYFE5ZUEZSR7LGZE5XBKMKY5YQ, seg_pos={0 0}, upload=477727]: Long tail cancellation due to optimal threshold Dec 28 08:14:48.397014129 (544h45m48.268097174s) [noise=true , address=45.91.116.14:4192 , piece_id=ARO7JTWXRAJ4CQ5TGIJ5JTW552MJPXVEUREJOM5CMVPECNBH33KQ, seg_pos={0 0}, upload=477727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:37844->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 08:14:48.397023076 (544h45m48.268106121s) [noise=true , address=45.91.116.14:4192 , piece_id=ARO7JTWXRAJ4CQ5TGIJ5JTW552MJPXVEUREJOM5CMVPECNBH33KQ, seg_pos={0 0}, upload=477727]: Long tail cancellation due to optimal threshold Dec 28 08:14:48.397267081 (544h45m48.268350126s) [noise=true , address=45.91.116.4:4939 , piece_id=G5XXE5S6TLE4BDXV3BDR2UVIOROKB3E5CUOUAUXQSNOBMEXTOEXA, seg_pos={0 0}, upload=477727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:40810->45.91.116.4:4939: use of closed network connection; context canceled Dec 28 08:14:48.397276288 (544h45m48.268359323s) [noise=true , address=45.91.116.4:4939 , piece_id=G5XXE5S6TLE4BDXV3BDR2UVIOROKB3E5CUOUAUXQSNOBMEXTOEXA, seg_pos={0 0}, upload=477727]: Long tail cancellation due to optimal threshold Dec 28 08:14:48.397293722 (544h45m48.268376767s) [seg_pos={0 0}, upload=477727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:14:48.397366931 (544h45m48.268449976s) [upload=477727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:14:48.397372792 (544h45m48.268455827s) [upload=477727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:14:48.695739013 (544h45m48.566822048s) [upload=477727]: Done uploading segments... Dec 28 08:14:48.695757137 (544h45m48.566840172s) [upload=477727]: Upload finished. err: <nil> Duration: 5.864282057s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)