Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [............X.X.....X.X.........X.X...XX......X.X..X..XX.XXX] duration: 5.001018433s trace: https://jaeger.dev.storj.io/trace/771aeae1520763d FAIL (2025-12-28 07:05:22 UTC) orbiter warning: context deadline exceeded Duration: 5.001204389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX............] duration: 5.992309387s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:04:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.992337702s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.............X.X.....X.X.........X.X...XX......X.X..X..XX.XX] duration: 5.001273305s trace: https://jaeger.dev.storj.io/trace/7e488dd16cb7aba FAIL (2025-12-28 07:04:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001295398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX...........] duration: 5.717163713s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.717190564s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..............X.X.....X.X.........X.X...XX......X.X..X..XX.X] duration: 5.00176656s trace: https://jaeger.dev.storj.io/trace/2f39ddcc1eed24cf FAIL (2025-12-28 07:03:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001830571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX..........] duration: 6.044167025s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:03:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 6.044191149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX.........] duration: 5.791803165s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:02:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.791856576s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...............X.X.....X.X.........X.X...XX......X.X..X..XX.] duration: 5.001790629s trace: https://jaeger.dev.storj.io/trace/150c20e47c056364 FAIL (2025-12-28 07:02:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001817491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX........] duration: 5.731999788s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:02:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.732025746s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [................X.X.....X.X.........X.X...XX......X.X..X..XX] duration: 5.001472629s trace: https://jaeger.dev.storj.io/trace/150c20e47c056364 FAIL (2025-12-28 07:01:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001494519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX.......] duration: 5.969310619s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:00:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.969350376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X................X.X.....X.X.........X.X...XX......X.X..X..X] duration: 5.001908891s trace: https://jaeger.dev.storj.io/trace/61ab74bc056dd632 FAIL (2025-12-28 07:00:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001934279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX......] duration: 6.0838399s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:00:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 6.083870197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX.....] duration: 6.074005587s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 07:00:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 6.074035945s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X................X.X.....X.X.........X.X...XX......X.X..X..] duration: 3.912832403s trace: https://jaeger.dev.storj.io/trace/60cf1ff2ad38c4a2 FAIL (2025-12-28 06:59:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.912859915s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X................X.X.....X.X.........X.X...XX......X.X..X.] duration: 5.001237704s trace: https://jaeger.dev.storj.io/trace/60cf1ff2ad38c4a2 FAIL (2025-12-28 06:59:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00126718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX....] duration: 5.670479848s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 06:58:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.670500687s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...X................X.X.....X.X.........X.X...XX......X.X..X] duration: 5.001145817s trace: https://jaeger.dev.storj.io/trace/60cf1ff2ad38c4a2 FAIL (2025-12-28 06:58:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001171837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX...] duration: 10.32729353s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 06:58:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 10.327312977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX..] duration: 5.940594427s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 06:57:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.940628643s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....X................X.X.....X.X.........X.X...XX......X.X..] duration: 3.456309896s trace: https://jaeger.dev.storj.io/trace/4facc4cc3228fc5f FAIL (2025-12-28 06:57:39 UTC) orbiter warning: context deadline exceeded Duration: 3.456336215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX.] duration: 5.917473068s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 06:57:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 5.917496683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....X................X.X.....X.X.........X.X...XX......X.X.] duration: 3.390026076s trace: https://jaeger.dev.storj.io/trace/4facc4cc3228fc5f FAIL (2025-12-28 06:57:09 UTC) orbiter warning: context deadline exceeded Duration: 3.390060933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.XX] duration: 30.002979694s trace: https://jaeger.dev.storj.io/trace/3f2f8e033c1788dc FAIL (2025-12-28 06:57:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA) context deadline exceeded ======LOGS====== 27m32.801645588s) [upload=476586]: Done starting upload Dec 28 06:56:32.930668264 (543h27m32.801751309s) [upload=476586]: Uploading segments... Dec 28 06:56:32.930801919 (543h27m32.801884954s) [upload=476586]: Got next segment. Inline: false Dec 28 06:56:32.930837927 (543h27m32.801920962s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:56:32.930844119 (543h27m32.801927154s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:56:33.173859216 (543h27m33.044942281s) [seg_pos={0 0}, upload=476586]: Begin upload segment... Dec 28 06:56:33.173918680 (543h27m33.045001725s) [seg_pos={0 0}, upload=476586]: Done begin upload segment. Dec 28 06:56:33.173974165 (543h27m33.04505721s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174014442 (543h27m33.045097487s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.174800809 (543h27m33.045883854s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175237110 (543h27m33.046320156s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.175833005 (543h27m33.04691605s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176318379 (543h27m33.047401414s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.176919704 (543h27m33.048002749s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.177485251 (543h27m33.048568286s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.178610624 (543h27m33.049693659s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.179339342 (543h27m33.050422387s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.180349596 (543h27m33.051432631s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181117398 (543h27m33.052200433s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.181688766 (543h27m33.052771801s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.185618258 (543h27m33.056701293s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.187031990 (543h27m33.058115035s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:33.192393230 (543h27m33.063476275s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Uploading piece... Dec 28 06:56:34.913370747 (543h27m34.784453782s) [upload=476586]: Next returned nil segment Dec 28 06:56:34.913384643 (543h27m34.784467678s) [upload=476586]: Waiting for error group managing segments... Dec 28 06:56:35.200131838 (543h27m35.071214883s) [noise=true , address=145.79.227.4:4185 , piece_id=A3C7RIAKGKHV7WC3UHD4Q5HTTDPMBAPGAUAKQFYVPU7UYTUFVYIA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.206905836 (543h27m35.077988881s) [noise=true , address=145.79.227.15:4876 , piece_id=P73AWGSYAQG4JNFM4A4AA2C5W7SQNXHNGLDMDQ4U32XGEO5H7EJQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.207904999 (543h27m35.078988034s) [noise=true , address=145.79.227.17:4971 , piece_id=DLEWW5V57BXEBLVQFHRN553RV7BBHSWU23H25Q22V2F3HBGR6YIQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.209284807 (543h27m35.080367852s) [noise=true , address=145.79.227.37:4565 , piece_id=BLMXT5WWH5QIR44ARSUA5W234EOS4IFCDERCJDTJ6UOYS3BBMANQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.216759219 (543h27m35.087842254s) [noise=true , address=145.79.227.36:4519 , piece_id=WD2K2JPXPKJDFC4PF3LIOUU6QBUWMBUNGP4KZSHCI4MK35RIK7QQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311344412 (543h27m35.182427457s) [noise=true , address=145.79.227.46:4602 , piece_id=OOUR7SXBIXW6VXHIDI7E4JF2DONJ5VXQXTVZROIMSMZQY656G7ZQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.311749884 (543h27m35.182832919s) [noise=true , address=145.79.227.50:4375 , piece_id=AMPGIVGAR7M63HAGICMDF5XDJGJZYEV7VZDU43RCTXS6QBUWY5VA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333360856 (543h27m35.204443892s) [noise=true , address=145.79.227.32:4835 , piece_id=2F7BCZFJ7AYP52KQCCBX4Q6FXPSJ7LRESMHMC2AFIXGNVHOJMISA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.333739878 (543h27m35.204822923s) [noise=true , address=145.79.227.19:4077 , piece_id=LWWGJ2ZLL56DZYGCXHWA5NW7A46ZFCZCKBE5XANL3D7VSPLFV6HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.342931922 (543h27m35.214014968s) [noise=true , address=145.79.227.13:4803 , piece_id=PTD76UPEHJJNY2WALHP3XO6BCTRUIWUIGNGH6KXOTTVPV5L2ZVQQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.429258578 (543h27m35.300341613s) [noise=true , address=45.91.116.4:4940 , piece_id=PSYEPYGZBS4BPL5CNVQCE6GKJ5JW6TBMZII4TY6IY3XIPVHXYVXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.457455001 (543h27m35.328538036s) [noise=true , address=77.90.31.10:4518 , piece_id=P4DRUG2LOR3T7XGDYKXEMVUA5KRF4JWISZ7PQLMLMUJJLSPP45DA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562801135 (543h27m35.43388418s) [noise=true , address=45.91.116.5:4299 , piece_id=YWWJ7KUGWF3EHT3O6KRSHU53P747LWWBE4BUJ45FHBN2VJMCW7KA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: <nil> Dec 28 06:56:35.562828416 (543h27m35.433911451s) [seg_pos={0 0}, upload=476586]: Segment reached optimal threshold of 13 pieces. Dec 28 06:56:35.563048166 (543h27m35.434131201s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48880->45.91.116.10:4081: use of closed network connection Dec 28 06:56:35.563056702 (543h27m35.434139737s) [noise=true , address=45.91.116.10:4081 , piece_id=B7HFSSYXME55F765VXQEKJR3NTV7N22WMKFLZMNAC5SM4LIYGLRA, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563193863 (543h27m35.434276908s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 06:56:35.563199423 (543h27m35.434282458s) [noise=true , address=145.79.227.49:4339 , piece_id=CHQARTV7GWDMMFYB7SNIVOB2JWYAY7HXO5HGUS4RRTESF2W5KSXQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563229430 (543h27m35.434312466s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:54690->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:56:35.563237936 (543h27m35.434320982s) [noise=true , address=45.91.116.13:4153 , piece_id=BPEZY5SSC5JUJXYOXY5V2PMQCZDUX7H4YUUI55LPUK6IDAXAA4HQ, seg_pos={0 0}, upload=476586]: Long tail cancellation due to optimal threshold Dec 28 06:56:35.563254559 (543h27m35.434337604s) [seg_pos={0 0}, upload=476586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:56:35.563335052 (543h27m35.434418087s) [upload=476586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:56:35.563340311 (543h27m35.434423357s) [upload=476586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:56:35.859038801 (543h27m35.730121846s) [upload=476586]: Done uploading segments... Dec 28 06:56:35.859065983 (543h27m35.730149048s) [upload=476586]: Upload finished. err: <nil> Duration: 30.003004943s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.....X................X.X.....X.X.........X.X...XX......X.X] duration: 5.000933402s trace: https://jaeger.dev.storj.io/trace/4facc4cc3228fc5f FAIL (2025-12-28 06:56:32 UTC) orbiter warning: context deadline exceeded Duration: 5.000957256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..X.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.X] duration: 30.003391914s trace: https://jaeger.dev.storj.io/trace/2e8a47b238bf5b4c FAIL (2025-12-28 06:56:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: QUN3BQK7I26DCLUGIO3GISAVO4XA7OQMPG4E5NMTNTBNXWASMOUA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 7FLRL36LMHUUW3PUGOSUSSLQMV5PTFRFEXOJHKP6VSDMI3FGL5FA) context deadline exceeded ======LOGS====== Dec 28 06:55:42.394302471 (543h26m42.265385516s) [upload=476574]: Starting upload Dec 28 06:55:42.394366814 (543h26m42.265449859s) [upload=476574]: Done starting upload Dec 28 06:55:42.394472365 (543h26m42.26555541s) [upload=476574]: Uploading segments... Dec 28 06:55:42.394595870 (543h26m42.265678915s) [upload=476574]: Got next segment. Inline: false Dec 28 06:55:42.394642469 (543h26m42.265725504s) [upload=476574]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:55:42.394647458 (543h26m42.265730503s) [upload=476574]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:55:42.642685962 (543h26m42.513769017s) [seg_pos={0 0}, upload=476574]: Begin upload segment... Dec 28 06:55:42.642739324 (543h26m42.513822379s) [seg_pos={0 0}, upload=476574]: Done begin upload segment. Dec 28 06:55:42.642781775 (543h26m42.51386482s) [noise=true , address=145.79.227.2:4113 , piece_id=HKVZ7YMOPM4MIOJ6DQULI6B2TQAGKSMF7BBX6JF64B5SLTBJTNFA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.642828483 (543h26m42.513911528s) [noise=true , address=145.79.227.45:4230 , piece_id=T7JHYTEOAVM7REELC7RKFVXRLIPJ7L2N2AMSOL56MHC34UCIOF7A, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.643722586 (543h26m42.514805631s) [noise=true , address=45.91.116.11:4118 , piece_id=QUN3BQK7I26DCLUGIO3GISAVO4XA7OQMPG4E5NMTNTBNXWASMOUA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.644236495 (543h26m42.51531954s) [noise=true , address=145.79.227.24:4583 , piece_id=4GGKXJHDIRVCI4ELJCY6QXYLPOGRPYGDISSMFEDPU7WIWKHIXGAQ, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.644674719 (543h26m42.515757764s) [noise=true , address=145.79.227.18:4007 , piece_id=5DWDJNTZNN6KJESVXCOJOI5DCRHEZOHUJJGSCCIMJI6URWNVXE7A, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.651311316 (543h26m42.522394362s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQRMHVZW62BI7FPP2LZAMVSVTGOQ2FAMTGO4PBNLZVTOF4Q2A4LQ, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.652856960 (543h26m42.523940005s) [noise=true , address=145.79.227.39:4013 , piece_id=TZMC7LT3RH5E5U3RK6PBO37ZLTTBXWJJWQW2GID7I7W7LKQAUBHA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.656923474 (543h26m42.528006519s) [noise=true , address=145.79.227.50:4375 , piece_id=4QXLRW4GPGI42QCOARNZWZ5E6KMKQGIAKD34WZDLHDWKHH4JM24Q, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.660577351 (543h26m42.531660397s) [noise=true , address=145.79.227.17:4972 , piece_id=U7A6O2FHL2DVXEJ6SY4VTYSSCYZ6M644N5OA6AMPGRCZI7HOOYTA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.663514916 (543h26m42.534597961s) [noise=true , address=45.91.116.4:4939 , piece_id=L6FWCC5DQL2VDYMMMUUQWMNR64GNGRE3KWN5ZLBSNFB5HUPKSHIA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.665404864 (543h26m42.536487909s) [noise=true , address=145.79.227.52:4446 , piece_id=KTDKZDLHPEEPVASFJJ64SHZR6ITCSXSKLBMP3OQSHN2HRAIFP2TA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.667008728 (543h26m42.538091773s) [noise=true , address=45.91.116.13:4153 , piece_id=7FLRL36LMHUUW3PUGOSUSSLQMV5PTFRFEXOJHKP6VSDMI3FGL5FA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.667787922 (543h26m42.538870968s) [noise=true , address=145.79.227.4:4186 , piece_id=KOQPMCJYYFZQ4R4IHFDJIN5VIKBYJ67NZKZIMP7MR4LAYEAJG7ZQ, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.669396796 (543h26m42.540479841s) [noise=true , address=145.79.227.40:4050 , piece_id=WLEN5QGBO3OLNPCI27IUSOK6UUTTCWMDN6PABA5IIRAPCH5ZWOXA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.670811209 (543h26m42.541894245s) [noise=true , address=145.79.227.3:4151 , piece_id=NPX2FME3KCHAAJUXEYM5N7SR7D7H5IIUVPU76UTAZDYUEDQ5GLUA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:42.671541901 (543h26m42.542624946s) [noise=true , address=145.79.227.46:4603 , piece_id=X7W3YY45TSBRBSYBZWMMD25UULNY2X4LNXHCXGUGNVA6OZPO3SQA, seg_pos={0 0}, upload=476574]: Uploading piece... Dec 28 06:55:44.354456706 (543h26m44.225539741s) [upload=476574]: Next returned nil segment Dec 28 06:55:44.354481814 (543h26m44.225564859s) [upload=476574]: Waiting for error group managing segments... Dec 28 06:55:44.637989268 (543h26m44.509072303s) [noise=true , address=145.79.227.2:4113 , piece_id=HKVZ7YMOPM4MIOJ6DQULI6B2TQAGKSMF7BBX6JF64B5SLTBJTNFA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.657267689 (543h26m44.528350724s) [noise=true , address=145.79.227.45:4230 , piece_id=T7JHYTEOAVM7REELC7RKFVXRLIPJ7L2N2AMSOL56MHC34UCIOF7A, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.662588130 (543h26m44.533671175s) [noise=true , address=145.79.227.18:4007 , piece_id=5DWDJNTZNN6KJESVXCOJOI5DCRHEZOHUJJGSCCIMJI6URWNVXE7A, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.664476196 (543h26m44.535559231s) [noise=true , address=145.79.227.4:4186 , piece_id=KOQPMCJYYFZQ4R4IHFDJIN5VIKBYJ67NZKZIMP7MR4LAYEAJG7ZQ, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.667123156 (543h26m44.538206191s) [noise=true , address=145.79.227.17:4972 , piece_id=U7A6O2FHL2DVXEJ6SY4VTYSSCYZ6M644N5OA6AMPGRCZI7HOOYTA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.670870953 (543h26m44.541954008s) [noise=true , address=145.79.227.46:4603 , piece_id=X7W3YY45TSBRBSYBZWMMD25UULNY2X4LNXHCXGUGNVA6OZPO3SQA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.673664142 (543h26m44.544747207s) [noise=true , address=145.79.227.24:4583 , piece_id=4GGKXJHDIRVCI4ELJCY6QXYLPOGRPYGDISSMFEDPU7WIWKHIXGAQ, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.683992039 (543h26m44.555075074s) [noise=true , address=145.79.227.52:4446 , piece_id=KTDKZDLHPEEPVASFJJ64SHZR6ITCSXSKLBMP3OQSHN2HRAIFP2TA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.685920250 (543h26m44.557003295s) [noise=true , address=145.79.227.50:4375 , piece_id=4QXLRW4GPGI42QCOARNZWZ5E6KMKQGIAKD34WZDLHDWKHH4JM24Q, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.686812660 (543h26m44.557895695s) [noise=true , address=145.79.227.39:4013 , piece_id=TZMC7LT3RH5E5U3RK6PBO37ZLTTBXWJJWQW2GID7I7W7LKQAUBHA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:44.867704269 (543h26m44.738787304s) [noise=true , address=45.91.116.11:4118 , piece_id=QUN3BQK7I26DCLUGIO3GISAVO4XA7OQMPG4E5NMTNTBNXWASMOUA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:45.488481656 (543h26m45.359564701s) [noise=true , address=145.79.227.3:4151 , piece_id=NPX2FME3KCHAAJUXEYM5N7SR7D7H5IIUVPU76UTAZDYUEDQ5GLUA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:45.600580177 (543h26m45.471663222s) [noise=true , address=45.91.116.13:4153 , piece_id=7FLRL36LMHUUW3PUGOSUSSLQMV5PTFRFEXOJHKP6VSDMI3FGL5FA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: <nil> Dec 28 06:55:45.600654519 (543h26m45.471737564s) [seg_pos={0 0}, upload=476574]: Segment reached optimal threshold of 13 pieces. Dec 28 06:55:45.600918281 (543h26m45.472001326s) [noise=true , address=45.91.116.4:4939 , piece_id=L6FWCC5DQL2VDYMMMUUQWMNR64GNGRE3KWN5ZLBSNFB5HUPKSHIA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 06:55:45.600979086 (543h26m45.472062131s) [noise=true , address=45.91.116.4:4939 , piece_id=L6FWCC5DQL2VDYMMMUUQWMNR64GNGRE3KWN5ZLBSNFB5HUPKSHIA, seg_pos={0 0}, upload=476574]: Long tail cancellation due to optimal threshold Dec 28 06:55:45.601142127 (543h26m45.472225172s) [noise=true , address=145.79.227.40:4050 , piece_id=WLEN5QGBO3OLNPCI27IUSOK6UUTTCWMDN6PABA5IIRAPCH5ZWOXA, seg_pos={0 0}, upload=476574]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 06:55:45.601161735 (543h26m45.47224478s) [noise=true , address=145.79.227.40:4050 , piece_id=WLEN5QGBO3OLNPCI27IUSOK6UUTTCWMDN6PABA5IIRAPCH5ZWOXA, seg_pos={0 0}, upload=476574]: Long tail cancellation due to optimal threshold Dec 28 06:55:45.601196361 (543h26m45.472279406s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQRMHVZW62BI7FPP2LZAMVSVTGOQ2FAMTGO4PBNLZVTOF4Q2A4LQ, seg_pos={0 0}, upload=476574]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 06:55:45.601207231 (543h26m45.472290266s) [noise=true , address=77.90.31.15:4697 , piece_id=ZQRMHVZW62BI7FPP2LZAMVSVTGOQ2FAMTGO4PBNLZVTOF4Q2A4LQ, seg_pos={0 0}, upload=476574]: Long tail cancellation due to optimal threshold Dec 28 06:55:45.601222912 (543h26m45.472305947s) [seg_pos={0 0}, upload=476574]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:55:45.601321559 (543h26m45.472404594s) [upload=476574]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:55:45.601327200 (543h26m45.472410245s) [upload=476574]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:55:45.903694258 (543h26m45.774777303s) [upload=476574]: Done uploading segments... Dec 28 06:55:45.903725718 (543h26m45.774808763s) [upload=476574]: Upload finished. err: <nil> Duration: 30.003417334s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XX.....X................X.X.....X.X.........X.X...XX......X.] duration: 4.415268658s trace: https://jaeger.dev.storj.io/trace/567312d95f17cd9 FAIL (2025-12-28 06:55:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.415298735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX.] duration: 5.790037444s trace: https://jaeger.dev.storj.io/trace/7796d53d39c777f4 FAIL (2025-12-28 06:54:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 55O22QGQAXWA24Q6LADGJUIOSJEPQV2IBZPARKOKBLPYTEUVQSPQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: A2NJS6ARF3OIRNVHGZJ2RTUQ773Y4CBMJW437P5OAAM26XJGLG2Q) context deadline exceeded ======LOGS====== Dec 28 06:54:05.471458264 (543h25m5.342541299s) [upload=476551]: Starting upload Dec 28 06:54:05.471580426 (543h25m5.342663491s) [upload=476551]: Done starting upload Dec 28 06:54:05.471679495 (543h25m5.34276253s) [upload=476551]: Uploading segments... Dec 28 06:54:05.471916086 (543h25m5.342999141s) [upload=476551]: Got next segment. Inline: false Dec 28 06:54:05.471970530 (543h25m5.343053595s) [upload=476551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:54:05.471977653 (543h25m5.343060708s) [upload=476551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:54:05.722944593 (543h25m5.594027668s) [seg_pos={0 0}, upload=476551]: Begin upload segment... Dec 28 06:54:05.723004698 (543h25m5.594087743s) [seg_pos={0 0}, upload=476551]: Done begin upload segment. Dec 28 06:54:05.723048631 (543h25m5.594131676s) [noise=true , address=145.79.227.29:4943 , piece_id=64CKMZ2UCL6OQWRULPHPQ3RTP7KDZ3P6MUA23SGTXVKNSLKXS6NQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.723062257 (543h25m5.594145302s) [noise=true , address=45.91.116.8:4047 , piece_id=BZ7JG4TDHJF3QQGVX5NGLCHPJYZTAUDOXUAWVJDOU3RAVYLWQV5Q, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.723416181 (543h25m5.594499226s) [noise=true , address=45.91.116.6:4976 , piece_id=YYRLHMYLIHBYE25YS5CWSM4PFOQKL5V6H4EPPCUAVE2RVHAFZWDA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.724267252 (543h25m5.595350297s) [noise=true , address=145.79.227.45:4231 , piece_id=VGN5DPTVWAUGZQPNPTXOEZGBKYTYDZBIEUDMW7FFWJRH7FMXJAZA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.724449198 (543h25m5.595532273s) [noise=true , address=145.79.227.4:4187 , piece_id=4SOI7FQI6QQ22PTWRTACF5R23ZKDSGE6YEKTCWZQZ2C2QVNRMXBQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.724878576 (543h25m5.595961641s) [noise=true , address=145.79.227.27:4691 , piece_id=HVRUDZTE4T6RM3VI32OLNTBMB4GSMVMGHNC2ABVZMVS5BO3EF2XA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.725078987 (543h25m5.596162032s) [noise=true , address=45.91.116.10:4082 , piece_id=IW4IU2CRRRAOSIRY6XVNAK2Y5TPITEHRNML3GS2RIF22Q5J54MMA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.725628544 (543h25m5.596711589s) [noise=true , address=45.91.116.13:4156 , piece_id=US4VYETX4QXA22EC3BS652JMVRBEZO3MNTDFQYEM3DJYZ7DAS5CA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.726531653 (543h25m5.597614698s) [noise=true , address=45.91.116.9:4262 , piece_id=55O22QGQAXWA24Q6LADGJUIOSJEPQV2IBZPARKOKBLPYTEUVQSPQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.727163887 (543h25m5.598246932s) [noise=true , address=145.79.227.11:4732 , piece_id=RUC2EZ7OQKFHL5PGWEZ75S4VD4CWYC3GEWPTOZHOFAXJDEQXX6IA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.727740045 (543h25m5.59882309s) [noise=true , address=145.79.227.15:4876 , piece_id=XI54FS5ARUTMCJBWQVONECFJ7QPTTHCYCEFS57QBJOLOZ4G27CXA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.728366888 (543h25m5.599449933s) [noise=true , address=77.90.31.19:4842 , piece_id=A2NJS6ARF3OIRNVHGZJ2RTUQ773Y4CBMJW437P5OAAM26XJGLG2Q, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.729006897 (543h25m5.600089943s) [noise=true , address=145.79.227.14:4838 , piece_id=XI4P65ISB5FUD5YIJS5A4JNIRYOWGT7XNLD6YJ5O4PJXUXKKWYAQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.729553397 (543h25m5.600636442s) [noise=true , address=145.79.227.48:4303 , piece_id=CZCGU7UJQKDZXIZRDZMOSNVY3RI6DIV2WAHSJJJ3EVALZR3HSSIQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.730240777 (543h25m5.601323822s) [noise=true , address=77.90.31.16:4736 , piece_id=W3KB4C37FIENMYXYWBIHATHNUEB2VPW7YJJDBBJACHS6NAVJG2IQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.730633484 (543h25m5.601716529s) [noise=true , address=145.79.227.31:4797 , piece_id=WYKY6DZ4E6E4FJESQPONXG62VGALUVKHAVSVO3UI6LJFIYTDQM6A, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:07.339323338 (543h25m7.210406383s) [upload=476551]: Next returned nil segment Dec 28 06:54:07.339345150 (543h25m7.210428215s) [upload=476551]: Waiting for error group managing segments... Dec 28 06:54:07.725789260 (543h25m7.596872306s) [noise=true , address=145.79.227.11:4732 , piece_id=RUC2EZ7OQKFHL5PGWEZ75S4VD4CWYC3GEWPTOZHOFAXJDEQXX6IA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.736403111 (543h25m7.607486156s) [noise=true , address=145.79.227.48:4303 , piece_id=CZCGU7UJQKDZXIZRDZMOSNVY3RI6DIV2WAHSJJJ3EVALZR3HSSIQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.738159546 (543h25m7.609242591s) [noise=true , address=145.79.227.45:4231 , piece_id=VGN5DPTVWAUGZQPNPTXOEZGBKYTYDZBIEUDMW7FFWJRH7FMXJAZA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.746708135 (543h25m7.61779117s) [noise=true , address=145.79.227.27:4691 , piece_id=HVRUDZTE4T6RM3VI32OLNTBMB4GSMVMGHNC2ABVZMVS5BO3EF2XA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.749182687 (543h25m7.620265732s) [noise=true , address=145.79.227.15:4876 , piece_id=XI54FS5ARUTMCJBWQVONECFJ7QPTTHCYCEFS57QBJOLOZ4G27CXA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.751502945 (543h25m7.62258598s) [noise=true , address=145.79.227.4:4187 , piece_id=4SOI7FQI6QQ22PTWRTACF5R23ZKDSGE6YEKTCWZQZ2C2QVNRMXBQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.751514467 (543h25m7.622597512s) [noise=true , address=145.79.227.14:4838 , piece_id=XI4P65ISB5FUD5YIJS5A4JNIRYOWGT7XNLD6YJ5O4PJXUXKKWYAQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.755476531 (543h25m7.626559576s) [noise=true , address=145.79.227.31:4797 , piece_id=WYKY6DZ4E6E4FJESQPONXG62VGALUVKHAVSVO3UI6LJFIYTDQM6A, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.760965133 (543h25m7.632048178s) [noise=true , address=145.79.227.29:4943 , piece_id=64CKMZ2UCL6OQWRULPHPQ3RTP7KDZ3P6MUA23SGTXVKNSLKXS6NQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.906157083 (543h25m7.777240128s) [noise=true , address=77.90.31.19:4842 , piece_id=A2NJS6ARF3OIRNVHGZJ2RTUQ773Y4CBMJW437P5OAAM26XJGLG2Q, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.981049575 (543h25m7.85213262s) [noise=true , address=45.91.116.10:4082 , piece_id=IW4IU2CRRRAOSIRY6XVNAK2Y5TPITEHRNML3GS2RIF22Q5J54MMA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.990922115 (543h25m7.86200516s) [noise=true , address=45.91.116.13:4156 , piece_id=US4VYETX4QXA22EC3BS652JMVRBEZO3MNTDFQYEM3DJYZ7DAS5CA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:08.006703637 (543h25m7.877786682s) [noise=true , address=45.91.116.9:4262 , piece_id=55O22QGQAXWA24Q6LADGJUIOSJEPQV2IBZPARKOKBLPYTEUVQSPQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:08.006749474 (543h25m7.877832519s) [seg_pos={0 0}, upload=476551]: Segment reached optimal threshold of 13 pieces. Dec 28 06:54:08.006970334 (543h25m7.878053379s) [noise=true , address=77.90.31.16:4736 , piece_id=W3KB4C37FIENMYXYWBIHATHNUEB2VPW7YJJDBBJACHS6NAVJG2IQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 06:54:08.006979132 (543h25m7.878062177s) [noise=true , address=77.90.31.16:4736 , piece_id=W3KB4C37FIENMYXYWBIHATHNUEB2VPW7YJJDBBJACHS6NAVJG2IQ, seg_pos={0 0}, upload=476551]: Long tail cancellation due to optimal threshold Dec 28 06:54:08.007166087 (543h25m7.878249133s) [noise=true , address=45.91.116.8:4047 , piece_id=BZ7JG4TDHJF3QQGVX5NGLCHPJYZTAUDOXUAWVJDOU3RAVYLWQV5Q, seg_pos={0 0}, upload=476551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 06:54:08.007174153 (543h25m7.878257198s) [noise=true , address=45.91.116.8:4047 , piece_id=BZ7JG4TDHJF3QQGVX5NGLCHPJYZTAUDOXUAWVJDOU3RAVYLWQV5Q, seg_pos={0 0}, upload=476551]: Long tail cancellation due to optimal threshold Dec 28 06:54:08.007236772 (543h25m7.878319817s) [noise=true , address=45.91.116.6:4976 , piece_id=YYRLHMYLIHBYE25YS5CWSM4PFOQKL5V6H4EPPCUAVE2RVHAFZWDA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 06:54:08.007461320 (543h25m7.878544365s) [noise=true , address=45.91.116.6:4976 , piece_id=YYRLHMYLIHBYE25YS5CWSM4PFOQKL5V6H4EPPCUAVE2RVHAFZWDA, seg_pos={0 0}, upload=476551]: Long tail cancellation due to optimal threshold Dec 28 06:54:08.007499282 (543h25m7.878582327s) [seg_pos={0 0}, upload=476551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:54:08.007607769 (543h25m7.878690814s) [upload=476551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:54:08.007615583 (543h25m7.878698629s) [upload=476551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:54:08.304559804 (543h25m8.175642849s) [upload=476551]: Done uploading segments... Dec 28 06:54:08.304581586 (543h25m8.175664631s) [upload=476551]: Upload finished. err: <nil> Duration: 5.790051991s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.XX.....X................X.X.....X.X.........X.X...XX......X] duration: 5.001540589s trace: https://jaeger.dev.storj.io/trace/567312d95f17cd9 FAIL (2025-12-28 06:54:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001561309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X...XXX] duration: 30.003181141s trace: https://jaeger.dev.storj.io/trace/7796d53d39c777f4 FAIL (2025-12-28 06:54:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 55O22QGQAXWA24Q6LADGJUIOSJEPQV2IBZPARKOKBLPYTEUVQSPQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: A2NJS6ARF3OIRNVHGZJ2RTUQ773Y4CBMJW437P5OAAM26XJGLG2Q) context deadline exceeded ======LOGS====== Dec 28 06:54:05.471458264 (543h25m5.342541299s) [upload=476551]: Starting upload Dec 28 06:54:05.471580426 (543h25m5.342663491s) [upload=476551]: Done starting upload Dec 28 06:54:05.471679495 (543h25m5.34276253s) [upload=476551]: Uploading segments... Dec 28 06:54:05.471916086 (543h25m5.342999141s) [upload=476551]: Got next segment. Inline: false Dec 28 06:54:05.471970530 (543h25m5.343053595s) [upload=476551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:54:05.471977653 (543h25m5.343060708s) [upload=476551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:54:05.722944593 (543h25m5.594027668s) [seg_pos={0 0}, upload=476551]: Begin upload segment... Dec 28 06:54:05.723004698 (543h25m5.594087743s) [seg_pos={0 0}, upload=476551]: Done begin upload segment. Dec 28 06:54:05.723048631 (543h25m5.594131676s) [noise=true , address=145.79.227.29:4943 , piece_id=64CKMZ2UCL6OQWRULPHPQ3RTP7KDZ3P6MUA23SGTXVKNSLKXS6NQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.723062257 (543h25m5.594145302s) [noise=true , address=45.91.116.8:4047 , piece_id=BZ7JG4TDHJF3QQGVX5NGLCHPJYZTAUDOXUAWVJDOU3RAVYLWQV5Q, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.723416181 (543h25m5.594499226s) [noise=true , address=45.91.116.6:4976 , piece_id=YYRLHMYLIHBYE25YS5CWSM4PFOQKL5V6H4EPPCUAVE2RVHAFZWDA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.724267252 (543h25m5.595350297s) [noise=true , address=145.79.227.45:4231 , piece_id=VGN5DPTVWAUGZQPNPTXOEZGBKYTYDZBIEUDMW7FFWJRH7FMXJAZA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.724449198 (543h25m5.595532273s) [noise=true , address=145.79.227.4:4187 , piece_id=4SOI7FQI6QQ22PTWRTACF5R23ZKDSGE6YEKTCWZQZ2C2QVNRMXBQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.724878576 (543h25m5.595961641s) [noise=true , address=145.79.227.27:4691 , piece_id=HVRUDZTE4T6RM3VI32OLNTBMB4GSMVMGHNC2ABVZMVS5BO3EF2XA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.725078987 (543h25m5.596162032s) [noise=true , address=45.91.116.10:4082 , piece_id=IW4IU2CRRRAOSIRY6XVNAK2Y5TPITEHRNML3GS2RIF22Q5J54MMA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.725628544 (543h25m5.596711589s) [noise=true , address=45.91.116.13:4156 , piece_id=US4VYETX4QXA22EC3BS652JMVRBEZO3MNTDFQYEM3DJYZ7DAS5CA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.726531653 (543h25m5.597614698s) [noise=true , address=45.91.116.9:4262 , piece_id=55O22QGQAXWA24Q6LADGJUIOSJEPQV2IBZPARKOKBLPYTEUVQSPQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.727163887 (543h25m5.598246932s) [noise=true , address=145.79.227.11:4732 , piece_id=RUC2EZ7OQKFHL5PGWEZ75S4VD4CWYC3GEWPTOZHOFAXJDEQXX6IA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.727740045 (543h25m5.59882309s) [noise=true , address=145.79.227.15:4876 , piece_id=XI54FS5ARUTMCJBWQVONECFJ7QPTTHCYCEFS57QBJOLOZ4G27CXA, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.728366888 (543h25m5.599449933s) [noise=true , address=77.90.31.19:4842 , piece_id=A2NJS6ARF3OIRNVHGZJ2RTUQ773Y4CBMJW437P5OAAM26XJGLG2Q, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.729006897 (543h25m5.600089943s) [noise=true , address=145.79.227.14:4838 , piece_id=XI4P65ISB5FUD5YIJS5A4JNIRYOWGT7XNLD6YJ5O4PJXUXKKWYAQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.729553397 (543h25m5.600636442s) [noise=true , address=145.79.227.48:4303 , piece_id=CZCGU7UJQKDZXIZRDZMOSNVY3RI6DIV2WAHSJJJ3EVALZR3HSSIQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.730240777 (543h25m5.601323822s) [noise=true , address=77.90.31.16:4736 , piece_id=W3KB4C37FIENMYXYWBIHATHNUEB2VPW7YJJDBBJACHS6NAVJG2IQ, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:05.730633484 (543h25m5.601716529s) [noise=true , address=145.79.227.31:4797 , piece_id=WYKY6DZ4E6E4FJESQPONXG62VGALUVKHAVSVO3UI6LJFIYTDQM6A, seg_pos={0 0}, upload=476551]: Uploading piece... Dec 28 06:54:07.339323338 (543h25m7.210406383s) [upload=476551]: Next returned nil segment Dec 28 06:54:07.339345150 (543h25m7.210428215s) [upload=476551]: Waiting for error group managing segments... Dec 28 06:54:07.725789260 (543h25m7.596872306s) [noise=true , address=145.79.227.11:4732 , piece_id=RUC2EZ7OQKFHL5PGWEZ75S4VD4CWYC3GEWPTOZHOFAXJDEQXX6IA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.736403111 (543h25m7.607486156s) [noise=true , address=145.79.227.48:4303 , piece_id=CZCGU7UJQKDZXIZRDZMOSNVY3RI6DIV2WAHSJJJ3EVALZR3HSSIQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.738159546 (543h25m7.609242591s) [noise=true , address=145.79.227.45:4231 , piece_id=VGN5DPTVWAUGZQPNPTXOEZGBKYTYDZBIEUDMW7FFWJRH7FMXJAZA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.746708135 (543h25m7.61779117s) [noise=true , address=145.79.227.27:4691 , piece_id=HVRUDZTE4T6RM3VI32OLNTBMB4GSMVMGHNC2ABVZMVS5BO3EF2XA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.749182687 (543h25m7.620265732s) [noise=true , address=145.79.227.15:4876 , piece_id=XI54FS5ARUTMCJBWQVONECFJ7QPTTHCYCEFS57QBJOLOZ4G27CXA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.751502945 (543h25m7.62258598s) [noise=true , address=145.79.227.4:4187 , piece_id=4SOI7FQI6QQ22PTWRTACF5R23ZKDSGE6YEKTCWZQZ2C2QVNRMXBQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.751514467 (543h25m7.622597512s) [noise=true , address=145.79.227.14:4838 , piece_id=XI4P65ISB5FUD5YIJS5A4JNIRYOWGT7XNLD6YJ5O4PJXUXKKWYAQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.755476531 (543h25m7.626559576s) [noise=true , address=145.79.227.31:4797 , piece_id=WYKY6DZ4E6E4FJESQPONXG62VGALUVKHAVSVO3UI6LJFIYTDQM6A, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.760965133 (543h25m7.632048178s) [noise=true , address=145.79.227.29:4943 , piece_id=64CKMZ2UCL6OQWRULPHPQ3RTP7KDZ3P6MUA23SGTXVKNSLKXS6NQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.906157083 (543h25m7.777240128s) [noise=true , address=77.90.31.19:4842 , piece_id=A2NJS6ARF3OIRNVHGZJ2RTUQ773Y4CBMJW437P5OAAM26XJGLG2Q, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.981049575 (543h25m7.85213262s) [noise=true , address=45.91.116.10:4082 , piece_id=IW4IU2CRRRAOSIRY6XVNAK2Y5TPITEHRNML3GS2RIF22Q5J54MMA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:07.990922115 (543h25m7.86200516s) [noise=true , address=45.91.116.13:4156 , piece_id=US4VYETX4QXA22EC3BS652JMVRBEZO3MNTDFQYEM3DJYZ7DAS5CA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:08.006703637 (543h25m7.877786682s) [noise=true , address=45.91.116.9:4262 , piece_id=55O22QGQAXWA24Q6LADGJUIOSJEPQV2IBZPARKOKBLPYTEUVQSPQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: <nil> Dec 28 06:54:08.006749474 (543h25m7.877832519s) [seg_pos={0 0}, upload=476551]: Segment reached optimal threshold of 13 pieces. Dec 28 06:54:08.006970334 (543h25m7.878053379s) [noise=true , address=77.90.31.16:4736 , piece_id=W3KB4C37FIENMYXYWBIHATHNUEB2VPW7YJJDBBJACHS6NAVJG2IQ, seg_pos={0 0}, upload=476551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 06:54:08.006979132 (543h25m7.878062177s) [noise=true , address=77.90.31.16:4736 , piece_id=W3KB4C37FIENMYXYWBIHATHNUEB2VPW7YJJDBBJACHS6NAVJG2IQ, seg_pos={0 0}, upload=476551]: Long tail cancellation due to optimal threshold Dec 28 06:54:08.007166087 (543h25m7.878249133s) [noise=true , address=45.91.116.8:4047 , piece_id=BZ7JG4TDHJF3QQGVX5NGLCHPJYZTAUDOXUAWVJDOU3RAVYLWQV5Q, seg_pos={0 0}, upload=476551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 06:54:08.007174153 (543h25m7.878257198s) [noise=true , address=45.91.116.8:4047 , piece_id=BZ7JG4TDHJF3QQGVX5NGLCHPJYZTAUDOXUAWVJDOU3RAVYLWQV5Q, seg_pos={0 0}, upload=476551]: Long tail cancellation due to optimal threshold Dec 28 06:54:08.007236772 (543h25m7.878319817s) [noise=true , address=45.91.116.6:4976 , piece_id=YYRLHMYLIHBYE25YS5CWSM4PFOQKL5V6H4EPPCUAVE2RVHAFZWDA, seg_pos={0 0}, upload=476551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 06:54:08.007461320 (543h25m7.878544365s) [noise=true , address=45.91.116.6:4976 , piece_id=YYRLHMYLIHBYE25YS5CWSM4PFOQKL5V6H4EPPCUAVE2RVHAFZWDA, seg_pos={0 0}, upload=476551]: Long tail cancellation due to optimal threshold Dec 28 06:54:08.007499282 (543h25m7.878582327s) [seg_pos={0 0}, upload=476551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:54:08.007607769 (543h25m7.878690814s) [upload=476551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:54:08.007615583 (543h25m7.878698629s) [upload=476551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:54:08.304559804 (543h25m8.175642849s) [upload=476551]: Done uploading segments... Dec 28 06:54:08.304581586 (543h25m8.175664631s) [upload=476551]: Upload finished. err: <nil> Duration: 30.003194327s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..XX.....X................X.X.....X.X.........X.X...XX......] duration: 5.00060665s trace: https://jaeger.dev.storj.io/trace/6ae0cf6d16a7134b FAIL (2025-12-28 06:54:18 UTC) orbiter warning: context deadline exceeded Duration: 5.00063297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X...XX] duration: 30.003020884s trace: https://jaeger.dev.storj.io/trace/6ca906c981a71ade FAIL (2025-12-28 06:54:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 06:53:35.182223462 (543h24m35.053306507s) [upload=476544]: Starting upload Dec 28 06:53:35.182280421 (543h24m35.053363456s) [upload=476544]: Done starting upload Dec 28 06:53:35.182346095 (543h24m35.05342913s) [upload=476544]: Uploading segments... Dec 28 06:53:35.182490511 (543h24m35.053573556s) [upload=476544]: Got next segment. Inline: false Dec 28 06:53:35.182520107 (543h24m35.053603142s) [upload=476544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:53:35.182524877 (543h24m35.053607922s) [upload=476544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:53:35.431932647 (543h24m35.303015692s) [seg_pos={0 0}, upload=476544]: Begin upload segment... Dec 28 06:53:35.431994094 (543h24m35.303077149s) [seg_pos={0 0}, upload=476544]: Done begin upload segment. Dec 28 06:53:35.432016837 (543h24m35.303099872s) [noise=true , address=145.79.227.15:4875 , piece_id=UAO4GBASX5LNETKCEO2FQQA2WWXOV7NOCL4CGVDBXWE7VIWOMJ4Q, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.432030793 (543h24m35.303113828s) [noise=true , address=145.79.227.27:4691 , piece_id=HUQO74CFO6RRQQZ2DKCI66FVUAV76EXZ4LVUGXUPKEPZ52SPGIEQ, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.432605478 (543h24m35.303688513s) [noise=true , address=77.90.31.18:4808 , piece_id=AODZFMNYY4QWX6PQWUB5BGFJLDENIKU6YAZWEP7HVQWSIFVYQTFQ, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.433520460 (543h24m35.304603505s) [noise=true , address=145.79.227.23:4547 , piece_id=GJRGH77TBN5YUMJW2ZL7AQ526Z2R25ZBL6M3JMBMYPJOF2M5IH3A, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.434025993 (543h24m35.305109048s) [noise=true , address=45.91.116.14:4190 , piece_id=BGHYX7VYH2VGMFBFJMISLT6M26SG3WZCWUK7WFRWCKSYXVXUTBAQ, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.434546004 (543h24m35.305629039s) [noise=true , address=45.91.116.5:4299 , piece_id=ZEYZKOBNDS2PANF6CE5QCBF23UQ3NUMLR4WX5C2IHHUJCM54IUCQ, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.434989188 (543h24m35.306072253s) [noise=true , address=145.79.227.24:4583 , piece_id=RXEKKYAGJI3A4BCJVMDV2HR7QCUCJCXCR5Z2QLL22LOUWYTYQSUA, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.435464462 (543h24m35.306547497s) [noise=true , address=145.79.227.39:4014 , piece_id=VIGUUMPHWCTMC5EE2Y2FSRKLP2LFCMNTUSUQ5WVOKI3AHPLE4DWA, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.436449148 (543h24m35.307532193s) [noise=true , address=145.79.227.6:4258 , piece_id=UE6OK3OXTWF4QACZCFHTD4PFJQ462OKIJL6E36Z5L4YHJZKKAU7A, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.437208583 (543h24m35.308291628s) [noise=true , address=45.91.116.7:4010 , piece_id=IRVIYG7DZJYCB25RHOKUF7NPQXLCDL2AWURPIGV5WIPU4P52PS7A, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.438038283 (543h24m35.309121328s) [noise=true , address=45.91.116.4:4940 , piece_id=JMORVXRSIDTYRJYOYX2GPJBWVMKLDM2USEVANUMM2LAVWGZBEPJQ, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.438883503 (543h24m35.309966538s) [noise=true , address=145.79.227.2:4115 , piece_id=53J4MYN6QI3R3BX5BQXZUUAXYUEEWQKBXJFYX7BHTTI5EUT4O7QA, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.439437608 (543h24m35.310520654s) [noise=true , address=145.79.227.5:4223 , piece_id=2D4KG4WY2UY3IYZAZGATUN666TK4K4552ANFZXF7PSAOVYUMEKVA, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.440477739 (543h24m35.311560774s) [noise=true , address=145.79.227.16:4909 , piece_id=K66N2VJYQRRRZ2TERG2GKGZ3HXRJ7RKZK3PFGXXIRUPOGOWNAO7A, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.442464091 (543h24m35.313547137s) [noise=true , address=145.79.227.45:4230 , piece_id=ESH3W3CWYWUWPZV2OC3QJWLD4QPOCJACMMVFS7HFXKOSLLZJ5GYA, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:35.445307045 (543h24m35.31639009s) [noise=true , address=145.79.227.13:4802 , piece_id=SZDL2NKCQGJFCOX3FALNA43BPXVZAZ64K4IUOAPDQUC4FKHDJ5XA, seg_pos={0 0}, upload=476544]: Uploading piece... Dec 28 06:53:37.149905942 (543h24m37.020989037s) [upload=476544]: Next returned nil segment Dec 28 06:53:37.149923706 (543h24m37.021006802s) [upload=476544]: Waiting for error group managing segments... Dec 28 06:53:37.444468007 (543h24m37.315551052s) [noise=true , address=145.79.227.15:4875 , piece_id=UAO4GBASX5LNETKCEO2FQQA2WWXOV7NOCL4CGVDBXWE7VIWOMJ4Q, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.445628237 (543h24m37.316711282s) [noise=true , address=145.79.227.13:4802 , piece_id=SZDL2NKCQGJFCOX3FALNA43BPXVZAZ64K4IUOAPDQUC4FKHDJ5XA, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.451129743 (543h24m37.322212788s) [noise=true , address=145.79.227.2:4115 , piece_id=53J4MYN6QI3R3BX5BQXZUUAXYUEEWQKBXJFYX7BHTTI5EUT4O7QA, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.451835788 (543h24m37.322918823s) [noise=true , address=145.79.227.5:4223 , piece_id=2D4KG4WY2UY3IYZAZGATUN666TK4K4552ANFZXF7PSAOVYUMEKVA, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.455690078 (543h24m37.326773113s) [noise=true , address=145.79.227.45:4230 , piece_id=ESH3W3CWYWUWPZV2OC3QJWLD4QPOCJACMMVFS7HFXKOSLLZJ5GYA, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.457350559 (543h24m37.328433594s) [noise=true , address=145.79.227.39:4014 , piece_id=VIGUUMPHWCTMC5EE2Y2FSRKLP2LFCMNTUSUQ5WVOKI3AHPLE4DWA, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.466766599 (543h24m37.337849634s) [noise=true , address=145.79.227.27:4691 , piece_id=HUQO74CFO6RRQQZ2DKCI66FVUAV76EXZ4LVUGXUPKEPZ52SPGIEQ, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.466891978 (543h24m37.337975013s) [noise=true , address=145.79.227.24:4583 , piece_id=RXEKKYAGJI3A4BCJVMDV2HR7QCUCJCXCR5Z2QLL22LOUWYTYQSUA, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.470851187 (543h24m37.341934222s) [noise=true , address=145.79.227.23:4547 , piece_id=GJRGH77TBN5YUMJW2ZL7AQ526Z2R25ZBL6M3JMBMYPJOF2M5IH3A, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.471298590 (543h24m37.342381625s) [noise=true , address=145.79.227.16:4909 , piece_id=K66N2VJYQRRRZ2TERG2GKGZ3HXRJ7RKZK3PFGXXIRUPOGOWNAO7A, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.480766228 (543h24m37.351849273s) [noise=true , address=145.79.227.6:4258 , piece_id=UE6OK3OXTWF4QACZCFHTD4PFJQ462OKIJL6E36Z5L4YHJZKKAU7A, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.647374760 (543h24m37.518457805s) [noise=true , address=77.90.31.18:4808 , piece_id=AODZFMNYY4QWX6PQWUB5BGFJLDENIKU6YAZWEP7HVQWSIFVYQTFQ, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.687349952 (543h24m37.558432987s) [noise=true , address=45.91.116.14:4190 , piece_id=BGHYX7VYH2VGMFBFJMISLT6M26SG3WZCWUK7WFRWCKSYXVXUTBAQ, seg_pos={0 0}, upload=476544]: Done uploading piece. err: <nil> Dec 28 06:53:37.687372064 (543h24m37.558455109s) [seg_pos={0 0}, upload=476544]: Segment reached optimal threshold of 13 pieces. Dec 28 06:53:37.688394722 (543h24m37.559477757s) [noise=true , address=45.91.116.4:4940 , piece_id=JMORVXRSIDTYRJYOYX2GPJBWVMKLDM2USEVANUMM2LAVWGZBEPJQ, seg_pos={0 0}, upload=476544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 06:53:37.688405822 (543h24m37.559488857s) [noise=true , address=45.91.116.4:4940 , piece_id=JMORVXRSIDTYRJYOYX2GPJBWVMKLDM2USEVANUMM2LAVWGZBEPJQ, seg_pos={0 0}, upload=476544]: Long tail cancellation due to optimal threshold Dec 28 06:53:37.688462090 (543h24m37.559545155s) [noise=true , address=45.91.116.7:4010 , piece_id=IRVIYG7DZJYCB25RHOKUF7NPQXLCDL2AWURPIGV5WIPU4P52PS7A, seg_pos={0 0}, upload=476544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 28 06:53:37.688478841 (543h24m37.559561886s) [noise=true , address=45.91.116.7:4010 , piece_id=IRVIYG7DZJYCB25RHOKUF7NPQXLCDL2AWURPIGV5WIPU4P52PS7A, seg_pos={0 0}, upload=476544]: Long tail cancellation due to optimal threshold Dec 28 06:53:37.688546290 (543h24m37.559629335s) [noise=true , address=45.91.116.5:4299 , piece_id=ZEYZKOBNDS2PANF6CE5QCBF23UQ3NUMLR4WX5C2IHHUJCM54IUCQ, seg_pos={0 0}, upload=476544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 06:53:37.688552772 (543h24m37.559635817s) [noise=true , address=45.91.116.5:4299 , piece_id=ZEYZKOBNDS2PANF6CE5QCBF23UQ3NUMLR4WX5C2IHHUJCM54IUCQ, seg_pos={0 0}, upload=476544]: Long tail cancellation due to optimal threshold Dec 28 06:53:37.688573462 (543h24m37.559656507s) [seg_pos={0 0}, upload=476544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:53:37.688668082 (543h24m37.559751137s) [upload=476544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:53:37.688673472 (543h24m37.559756527s) [upload=476544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:53:37.978125296 (543h24m37.849208341s) [upload=476544]: Done uploading segments... Dec 28 06:53:37.978152578 (543h24m37.849235623s) [upload=476544]: Upload finished. err: <nil> Duration: 30.003049529s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...XX.....X................X.X.....X.X.........X.X...XX.....] duration: 3.373361053s trace: https://jaeger.dev.storj.io/trace/6ae0cf6d16a7134b FAIL (2025-12-28 06:54:01 UTC) orbiter warning: context deadline exceeded Duration: 3.373390599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X...XX.....X................X.X.....X.X.........X.X...XX....] duration: 3.780297593s trace: https://jaeger.dev.storj.io/trace/6ae0cf6d16a7134b FAIL (2025-12-28 06:53:32 UTC) orbiter warning: context deadline exceeded Duration: 3.780326127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X...X] duration: 30.003511949s trace: https://jaeger.dev.storj.io/trace/38b17d5b62b5784f FAIL (2025-12-28 06:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: A5U5V66JI3RSIKEC7IU5XWPJGIZTOPUZ5WYRW7AVSTVLVR5XTEMA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: DBR3LYK2WRIU2JD5EXD3QW67GXXNKQF75RPNTKAEIVFE5L7TPD2Q) context deadline exceeded ======LOGS====== Dec 28 06:52:54.020728856 (543h23m53.891811891s) [upload=476534]: Starting upload Dec 28 06:52:54.020800482 (543h23m53.891883528s) [upload=476534]: Done starting upload Dec 28 06:52:54.020890354 (543h23m53.891973409s) [upload=476534]: Uploading segments... Dec 28 06:52:54.021006054 (543h23m53.892089099s) [upload=476534]: Got next segment. Inline: false Dec 28 06:52:54.021052383 (543h23m53.892135428s) [upload=476534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:52:54.021057112 (543h23m53.892140147s) [upload=476534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:52:54.264447474 (543h23m54.135530519s) [seg_pos={0 0}, upload=476534]: Begin upload segment... Dec 28 06:52:54.264511866 (543h23m54.135594911s) [seg_pos={0 0}, upload=476534]: Done begin upload segment. Dec 28 06:52:54.264546632 (543h23m54.135629697s) [noise=true , address=145.79.227.38:4979 , piece_id=3JKTKTWRY62ZJQEXUKS5HNXATFLC2SCFMCD4S5FQDKJA2VUD2BZQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.264564126 (543h23m54.135647171s) [noise=true , address=145.79.227.19:4078 , piece_id=V55CTESC3WQTXXDTA5NYPN3FB3UMICHJ2XZ6ORBUDKZY42IPLTZA, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.264613149 (543h23m54.135696244s) [noise=true , address=145.79.227.51:4411 , piece_id=V3WJ6H4QQTGDP6NHHSUED5YPK5GHH3VRVHJLPMDRFFCE53TM2DAQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.265479659 (543h23m54.136562704s) [noise=true , address=145.79.227.34:4905 , piece_id=MN3COMJJYW7236QQ5ESPDJHLVTX3E6NFKQTJF2KWYE7JBYSLLEKA, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.265976025 (543h23m54.13705907s) [noise=true , address=77.90.31.13:4629 , piece_id=DBR3LYK2WRIU2JD5EXD3QW67GXXNKQF75RPNTKAEIVFE5L7TPD2Q, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.266568884 (543h23m54.137651929s) [noise=true , address=145.79.227.15:4873 , piece_id=EMHDRUMU7J7FOBJDD5EEWBNDTJD3LXHION3WXH663UNQY5EMRGHQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.267046793 (543h23m54.138129838s) [noise=true , address=77.90.31.15:4700 , piece_id=J7SGPFQHWJ3FCHNJMILBT6X4OQRJRGKXYM4PBAGAXCHWEK53CL3Q, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.267524473 (543h23m54.138607518s) [noise=true , address=145.79.227.14:4839 , piece_id=MKWB2IFAMKYKR3U4SKQUBHUJ6V4RS2EAVF3ITP2I35A43N3MCHAQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.268741400 (543h23m54.139824435s) [noise=true , address=145.79.227.25:4619 , piece_id=IW23Z3DXYOTK25FJJ36DNHZ3DWM7KNPBLWC3UOAR7ZHAIZNT4RUA, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.269162692 (543h23m54.140245727s) [noise=true , address=45.91.116.5:4299 , piece_id=A5U5V66JI3RSIKEC7IU5XWPJGIZTOPUZ5WYRW7AVSTVLVR5XTEMA, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.270294488 (543h23m54.141377533s) [noise=true , address=45.91.116.13:4155 , piece_id=ZIPO3MSZPL74EQ6HXAYMXGH5YSBJDBFVSCBYQBZFCOQO2RAPR7UA, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.271436583 (543h23m54.142519628s) [noise=true , address=145.79.227.17:4970 , piece_id=BHLIWCDUADMGY33ERUMZP6O6SSBACEKYA233T6YJP6ZK4CYJJ6JA, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.271752134 (543h23m54.142835179s) [noise=true , address=145.79.227.31:4798 , piece_id=WRKEZCG5QOPS2H5Z2MC4Y2TKL26UR2TMVZK2EIIF54JHQVF42NMQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.274213150 (543h23m54.145296195s) [noise=true , address=45.91.116.12:4225 , piece_id=UYAKTBIAD5VKVPC3TOIERAFJMICMRRLJGSUC5WXW4TO2WRK3CUZQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.275903218 (543h23m54.146986263s) [noise=true , address=145.79.227.33:4871 , piece_id=TRCGVOSNJMV3FYKHB7JHT2UCPQ2R5K75VQA76ZMPSKJIXZU5B5RQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:54.279545003 (543h23m54.150628038s) [noise=true , address=145.79.227.47:4267 , piece_id=B3BNBY24CP3SEUWOMBNYPC7PIO6APZYV2MNQGOMSZTTUFCGJTXRQ, seg_pos={0 0}, upload=476534]: Uploading piece... Dec 28 06:52:55.989013921 (543h23m55.860096966s) [upload=476534]: Next returned nil segment Dec 28 06:52:55.989031415 (543h23m55.86011446s) [upload=476534]: Waiting for error group managing segments... Dec 28 06:52:56.272509894 (543h23m56.143592929s) [noise=true , address=145.79.227.51:4411 , piece_id=V3WJ6H4QQTGDP6NHHSUED5YPK5GHH3VRVHJLPMDRFFCE53TM2DAQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.285949737 (543h23m56.157032772s) [noise=true , address=145.79.227.14:4839 , piece_id=MKWB2IFAMKYKR3U4SKQUBHUJ6V4RS2EAVF3ITP2I35A43N3MCHAQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.286803072 (543h23m56.157886117s) [noise=true , address=145.79.227.15:4873 , piece_id=EMHDRUMU7J7FOBJDD5EEWBNDTJD3LXHION3WXH663UNQY5EMRGHQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.291873897 (543h23m56.162956953s) [noise=true , address=145.79.227.38:4979 , piece_id=3JKTKTWRY62ZJQEXUKS5HNXATFLC2SCFMCD4S5FQDKJA2VUD2BZQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.295373913 (543h23m56.166456958s) [noise=true , address=145.79.227.25:4619 , piece_id=IW23Z3DXYOTK25FJJ36DNHZ3DWM7KNPBLWC3UOAR7ZHAIZNT4RUA, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.304868102 (543h23m56.175951137s) [noise=true , address=145.79.227.19:4078 , piece_id=V55CTESC3WQTXXDTA5NYPN3FB3UMICHJ2XZ6ORBUDKZY42IPLTZA, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.305502210 (543h23m56.176585255s) [noise=true , address=145.79.227.31:4798 , piece_id=WRKEZCG5QOPS2H5Z2MC4Y2TKL26UR2TMVZK2EIIF54JHQVF42NMQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.307799835 (543h23m56.17888287s) [noise=true , address=145.79.227.17:4970 , piece_id=BHLIWCDUADMGY33ERUMZP6O6SSBACEKYA233T6YJP6ZK4CYJJ6JA, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.319206024 (543h23m56.190289069s) [noise=true , address=145.79.227.47:4267 , piece_id=B3BNBY24CP3SEUWOMBNYPC7PIO6APZYV2MNQGOMSZTTUFCGJTXRQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.466663789 (543h23m56.337746834s) [noise=true , address=77.90.31.15:4700 , piece_id=J7SGPFQHWJ3FCHNJMILBT6X4OQRJRGKXYM4PBAGAXCHWEK53CL3Q, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:56.651698792 (543h23m56.522781827s) [noise=true , address=77.90.31.13:4629 , piece_id=DBR3LYK2WRIU2JD5EXD3QW67GXXNKQF75RPNTKAEIVFE5L7TPD2Q, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:57.191597224 (543h23m57.062680269s) [noise=true , address=45.91.116.12:4225 , piece_id=UYAKTBIAD5VKVPC3TOIERAFJMICMRRLJGSUC5WXW4TO2WRK3CUZQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:57.448110927 (543h23m57.319193972s) [noise=true , address=45.91.116.5:4299 , piece_id=A5U5V66JI3RSIKEC7IU5XWPJGIZTOPUZ5WYRW7AVSTVLVR5XTEMA, seg_pos={0 0}, upload=476534]: Done uploading piece. err: <nil> Dec 28 06:52:57.448135433 (543h23m57.319218468s) [seg_pos={0 0}, upload=476534]: Segment reached optimal threshold of 13 pieces. Dec 28 06:52:57.448348499 (543h23m57.319431554s) [noise=true , address=45.91.116.13:4155 , piece_id=ZIPO3MSZPL74EQ6HXAYMXGH5YSBJDBFVSCBYQBZFCOQO2RAPR7UA, seg_pos={0 0}, upload=476534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 06:52:57.448359891 (543h23m57.319442936s) [noise=true , address=45.91.116.13:4155 , piece_id=ZIPO3MSZPL74EQ6HXAYMXGH5YSBJDBFVSCBYQBZFCOQO2RAPR7UA, seg_pos={0 0}, upload=476534]: Long tail cancellation due to optimal threshold Dec 28 06:52:57.448670513 (543h23m57.319753558s) [noise=true , address=145.79.227.33:4871 , piece_id=TRCGVOSNJMV3FYKHB7JHT2UCPQ2R5K75VQA76ZMPSKJIXZU5B5RQ, seg_pos={0 0}, upload=476534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 28 06:52:57.448679169 (543h23m57.319762224s) [noise=true , address=145.79.227.33:4871 , piece_id=TRCGVOSNJMV3FYKHB7JHT2UCPQ2R5K75VQA76ZMPSKJIXZU5B5RQ, seg_pos={0 0}, upload=476534]: Long tail cancellation due to optimal threshold Dec 28 06:52:57.448825377 (543h23m57.319908422s) [noise=true , address=145.79.227.34:4905 , piece_id=MN3COMJJYW7236QQ5ESPDJHLVTX3E6NFKQTJF2KWYE7JBYSLLEKA, seg_pos={0 0}, upload=476534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 06:52:57.448833603 (543h23m57.319916638s) [noise=true , address=145.79.227.34:4905 , piece_id=MN3COMJJYW7236QQ5ESPDJHLVTX3E6NFKQTJF2KWYE7JBYSLLEKA, seg_pos={0 0}, upload=476534]: Long tail cancellation due to optimal threshold Dec 28 06:52:57.448858731 (543h23m57.319941766s) [seg_pos={0 0}, upload=476534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:52:57.448944855 (543h23m57.3200279s) [upload=476534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:52:57.448952129 (543h23m57.320035174s) [upload=476534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:52:57.742389442 (543h23m57.613472487s) [upload=476534]: Done uploading segments... Dec 28 06:52:57.742411603 (543h23m57.613494638s) [upload=476534]: Upload finished. err: <nil> Duration: 30.003537457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X...XX.....X................X.X.....X.X.........X.X...XX...] duration: 3.350837314s trace: https://jaeger.dev.storj.io/trace/6ae0cf6d16a7134b FAIL (2025-12-28 06:52:23 UTC) orbiter warning: context deadline exceeded Duration: 3.350857091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X.X...XX.....X................X.X.....X.X.........X.X...XX..] duration: 4.518093944s trace: https://jaeger.dev.storj.io/trace/6ae0cf6d16a7134b FAIL (2025-12-28 06:52:06 UTC) orbiter warning: context deadline exceeded Duration: 4.518122248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X...] duration: 5.743759924s trace: https://jaeger.dev.storj.io/trace/146989eab2ea58c7 FAIL (2025-12-28 06:51:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ) context deadline exceeded ======LOGS====== load Dec 28 06:49:59.297128816 (543h20m59.168211871s) [upload=476491]: Done starting upload Dec 28 06:49:59.297215831 (543h20m59.168298886s) [upload=476491]: Uploading segments... Dec 28 06:49:59.297449166 (543h20m59.168532211s) [upload=476491]: Got next segment. Inline: false Dec 28 06:49:59.297510743 (543h20m59.168593798s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:49:59.297516845 (543h20m59.16859988s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:49:59.540204673 (543h20m59.411287719s) [seg_pos={0 0}, upload=476491]: Begin upload segment... Dec 28 06:49:59.540254087 (543h20m59.411337122s) [seg_pos={0 0}, upload=476491]: Done begin upload segment. Dec 28 06:49:59.540280478 (543h20m59.411363513s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540328299 (543h20m59.411411334s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540988667 (543h20m59.412071712s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541438833 (543h20m59.412521878s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541933876 (543h20m59.413016912s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542494925 (543h20m59.41357796s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542977033 (543h20m59.414060068s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542997411 (543h20m59.414080446s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543381653 (543h20m59.414464688s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543727081 (543h20m59.414810116s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544084831 (543h20m59.415167877s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544482139 (543h20m59.415565174s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544826845 (543h20m59.41590988s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.546119346 (543h20m59.417202391s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.551669674 (543h20m59.42275272s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.552830656 (543h20m59.423913691s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:50:01.258365217 (543h21m1.129448292s) [upload=476491]: Next returned nil segment Dec 28 06:50:01.258398810 (543h21m1.129481855s) [upload=476491]: Waiting for error group managing segments... Dec 28 06:50:01.543372188 (543h21m1.414455233s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.565558916 (543h21m1.436641971s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.576701373 (543h21m1.447784438s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.579147681 (543h21m1.450230726s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.585783266 (543h21m1.456866311s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.587092168 (543h21m1.458175213s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.588867348 (543h21m1.459950403s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591090392 (543h21m1.462173437s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591242731 (543h21m1.462325776s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.718671558 (543h21m1.589754603s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.786458726 (543h21m1.657541771s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.954373975 (543h21m1.82545706s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201054377 (543h21m2.072137432s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201346554 (543h21m2.072429609s) [seg_pos={0 0}, upload=476491]: Segment reached optimal threshold of 13 pieces. Dec 28 06:50:02.201691010 (543h21m2.072774055s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection Dec 28 06:50:02.201702451 (543h21m2.072785497s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202552920 (543h21m2.073635966s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 06:50:02.202565415 (543h21m2.07364846s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202783540 (543h21m2.073866595s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:34022->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:50:02.202796154 (543h21m2.073879199s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202823887 (543h21m2.073906932s) [seg_pos={0 0}, upload=476491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:50:02.202945759 (543h21m2.074028804s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:50:02.202954987 (543h21m2.074038032s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:50:02.492076845 (543h21m2.36315989s) [upload=476491]: Done uploading segments... Dec 28 06:50:02.492103696 (543h21m2.363186751s) [upload=476491]: Upload finished. err: <nil> Duration: 5.743789481s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX.X...XX.....X................X.X.....X.X.........X.X...XX.] duration: 3.422878337s trace: https://jaeger.dev.storj.io/trace/6ae0cf6d16a7134b FAIL (2025-12-28 06:51:28 UTC) orbiter warning: context deadline exceeded Duration: 3.422894939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X..] duration: 5.899267787s trace: https://jaeger.dev.storj.io/trace/146989eab2ea58c7 FAIL (2025-12-28 06:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ) context deadline exceeded ======LOGS====== load Dec 28 06:49:59.297128816 (543h20m59.168211871s) [upload=476491]: Done starting upload Dec 28 06:49:59.297215831 (543h20m59.168298886s) [upload=476491]: Uploading segments... Dec 28 06:49:59.297449166 (543h20m59.168532211s) [upload=476491]: Got next segment. Inline: false Dec 28 06:49:59.297510743 (543h20m59.168593798s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:49:59.297516845 (543h20m59.16859988s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:49:59.540204673 (543h20m59.411287719s) [seg_pos={0 0}, upload=476491]: Begin upload segment... Dec 28 06:49:59.540254087 (543h20m59.411337122s) [seg_pos={0 0}, upload=476491]: Done begin upload segment. Dec 28 06:49:59.540280478 (543h20m59.411363513s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540328299 (543h20m59.411411334s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540988667 (543h20m59.412071712s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541438833 (543h20m59.412521878s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541933876 (543h20m59.413016912s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542494925 (543h20m59.41357796s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542977033 (543h20m59.414060068s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542997411 (543h20m59.414080446s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543381653 (543h20m59.414464688s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543727081 (543h20m59.414810116s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544084831 (543h20m59.415167877s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544482139 (543h20m59.415565174s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544826845 (543h20m59.41590988s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.546119346 (543h20m59.417202391s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.551669674 (543h20m59.42275272s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.552830656 (543h20m59.423913691s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:50:01.258365217 (543h21m1.129448292s) [upload=476491]: Next returned nil segment Dec 28 06:50:01.258398810 (543h21m1.129481855s) [upload=476491]: Waiting for error group managing segments... Dec 28 06:50:01.543372188 (543h21m1.414455233s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.565558916 (543h21m1.436641971s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.576701373 (543h21m1.447784438s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.579147681 (543h21m1.450230726s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.585783266 (543h21m1.456866311s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.587092168 (543h21m1.458175213s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.588867348 (543h21m1.459950403s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591090392 (543h21m1.462173437s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591242731 (543h21m1.462325776s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.718671558 (543h21m1.589754603s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.786458726 (543h21m1.657541771s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.954373975 (543h21m1.82545706s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201054377 (543h21m2.072137432s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201346554 (543h21m2.072429609s) [seg_pos={0 0}, upload=476491]: Segment reached optimal threshold of 13 pieces. Dec 28 06:50:02.201691010 (543h21m2.072774055s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection Dec 28 06:50:02.201702451 (543h21m2.072785497s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202552920 (543h21m2.073635966s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 06:50:02.202565415 (543h21m2.07364846s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202783540 (543h21m2.073866595s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:34022->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:50:02.202796154 (543h21m2.073879199s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202823887 (543h21m2.073906932s) [seg_pos={0 0}, upload=476491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:50:02.202945759 (543h21m2.074028804s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:50:02.202954987 (543h21m2.074038032s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:50:02.492076845 (543h21m2.36315989s) [upload=476491]: Done uploading segments... Dec 28 06:50:02.492103696 (543h21m2.363186751s) [upload=476491]: Upload finished. err: <nil> Duration: 5.899291352s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.XX.X...XX.....X................X.X.....X.X.........X.X...XX] duration: 5.002114232s trace: https://jaeger.dev.storj.io/trace/6ae0cf6d16a7134b FAIL (2025-12-28 06:51:01 UTC) orbiter warning: context deadline exceeded Duration: 5.002141666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X.] duration: 5.985235816s trace: https://jaeger.dev.storj.io/trace/146989eab2ea58c7 FAIL (2025-12-28 06:50:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ) context deadline exceeded ======LOGS====== load Dec 28 06:49:59.297128816 (543h20m59.168211871s) [upload=476491]: Done starting upload Dec 28 06:49:59.297215831 (543h20m59.168298886s) [upload=476491]: Uploading segments... Dec 28 06:49:59.297449166 (543h20m59.168532211s) [upload=476491]: Got next segment. Inline: false Dec 28 06:49:59.297510743 (543h20m59.168593798s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:49:59.297516845 (543h20m59.16859988s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:49:59.540204673 (543h20m59.411287719s) [seg_pos={0 0}, upload=476491]: Begin upload segment... Dec 28 06:49:59.540254087 (543h20m59.411337122s) [seg_pos={0 0}, upload=476491]: Done begin upload segment. Dec 28 06:49:59.540280478 (543h20m59.411363513s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540328299 (543h20m59.411411334s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540988667 (543h20m59.412071712s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541438833 (543h20m59.412521878s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541933876 (543h20m59.413016912s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542494925 (543h20m59.41357796s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542977033 (543h20m59.414060068s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542997411 (543h20m59.414080446s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543381653 (543h20m59.414464688s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543727081 (543h20m59.414810116s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544084831 (543h20m59.415167877s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544482139 (543h20m59.415565174s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544826845 (543h20m59.41590988s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.546119346 (543h20m59.417202391s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.551669674 (543h20m59.42275272s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.552830656 (543h20m59.423913691s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:50:01.258365217 (543h21m1.129448292s) [upload=476491]: Next returned nil segment Dec 28 06:50:01.258398810 (543h21m1.129481855s) [upload=476491]: Waiting for error group managing segments... Dec 28 06:50:01.543372188 (543h21m1.414455233s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.565558916 (543h21m1.436641971s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.576701373 (543h21m1.447784438s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.579147681 (543h21m1.450230726s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.585783266 (543h21m1.456866311s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.587092168 (543h21m1.458175213s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.588867348 (543h21m1.459950403s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591090392 (543h21m1.462173437s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591242731 (543h21m1.462325776s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.718671558 (543h21m1.589754603s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.786458726 (543h21m1.657541771s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.954373975 (543h21m1.82545706s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201054377 (543h21m2.072137432s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201346554 (543h21m2.072429609s) [seg_pos={0 0}, upload=476491]: Segment reached optimal threshold of 13 pieces. Dec 28 06:50:02.201691010 (543h21m2.072774055s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection Dec 28 06:50:02.201702451 (543h21m2.072785497s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202552920 (543h21m2.073635966s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 06:50:02.202565415 (543h21m2.07364846s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202783540 (543h21m2.073866595s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:34022->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:50:02.202796154 (543h21m2.073879199s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202823887 (543h21m2.073906932s) [seg_pos={0 0}, upload=476491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:50:02.202945759 (543h21m2.074028804s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:50:02.202954987 (543h21m2.074038032s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:50:02.492076845 (543h21m2.36315989s) [upload=476491]: Done uploading segments... Dec 28 06:50:02.492103696 (543h21m2.363186751s) [upload=476491]: Upload finished. err: <nil> Duration: 5.985254661s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX.X...XX.....X................X.X.....X.X.........X.X...X] duration: 5.000814338s trace: https://jaeger.dev.storj.io/trace/662c07037cb3ac40 FAIL (2025-12-28 06:50:33 UTC) orbiter warning: context deadline exceeded Duration: 5.000918948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X] duration: 30.003951326s trace: https://jaeger.dev.storj.io/trace/146989eab2ea58c7 FAIL (2025-12-28 06:50:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ) context deadline exceeded ======LOGS====== load Dec 28 06:49:59.297128816 (543h20m59.168211871s) [upload=476491]: Done starting upload Dec 28 06:49:59.297215831 (543h20m59.168298886s) [upload=476491]: Uploading segments... Dec 28 06:49:59.297449166 (543h20m59.168532211s) [upload=476491]: Got next segment. Inline: false Dec 28 06:49:59.297510743 (543h20m59.168593798s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:49:59.297516845 (543h20m59.16859988s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:49:59.540204673 (543h20m59.411287719s) [seg_pos={0 0}, upload=476491]: Begin upload segment... Dec 28 06:49:59.540254087 (543h20m59.411337122s) [seg_pos={0 0}, upload=476491]: Done begin upload segment. Dec 28 06:49:59.540280478 (543h20m59.411363513s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540328299 (543h20m59.411411334s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.540988667 (543h20m59.412071712s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541438833 (543h20m59.412521878s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.541933876 (543h20m59.413016912s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542494925 (543h20m59.41357796s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542977033 (543h20m59.414060068s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.542997411 (543h20m59.414080446s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543381653 (543h20m59.414464688s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.543727081 (543h20m59.414810116s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544084831 (543h20m59.415167877s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544482139 (543h20m59.415565174s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.544826845 (543h20m59.41590988s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.546119346 (543h20m59.417202391s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.551669674 (543h20m59.42275272s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:49:59.552830656 (543h20m59.423913691s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Uploading piece... Dec 28 06:50:01.258365217 (543h21m1.129448292s) [upload=476491]: Next returned nil segment Dec 28 06:50:01.258398810 (543h21m1.129481855s) [upload=476491]: Waiting for error group managing segments... Dec 28 06:50:01.543372188 (543h21m1.414455233s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.565558916 (543h21m1.436641971s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.576701373 (543h21m1.447784438s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.579147681 (543h21m1.450230726s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.585783266 (543h21m1.456866311s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.587092168 (543h21m1.458175213s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.588867348 (543h21m1.459950403s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591090392 (543h21m1.462173437s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.591242731 (543h21m1.462325776s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.718671558 (543h21m1.589754603s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.786458726 (543h21m1.657541771s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:01.954373975 (543h21m1.82545706s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201054377 (543h21m2.072137432s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil> Dec 28 06:50:02.201346554 (543h21m2.072429609s) [seg_pos={0 0}, upload=476491]: Segment reached optimal threshold of 13 pieces. Dec 28 06:50:02.201691010 (543h21m2.072774055s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection Dec 28 06:50:02.201702451 (543h21m2.072785497s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202552920 (543h21m2.073635966s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 06:50:02.202565415 (543h21m2.07364846s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202783540 (543h21m2.073866595s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:34022->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:50:02.202796154 (543h21m2.073879199s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold Dec 28 06:50:02.202823887 (543h21m2.073906932s) [seg_pos={0 0}, upload=476491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:50:02.202945759 (543h21m2.074028804s) [upload=476491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:50:02.202954987 (543h21m2.074038032s) [upload=476491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:50:02.492076845 (543h21m2.36315989s) [upload=476491]: Done uploading segments... Dec 28 06:50:02.492103696 (543h21m2.363186751s) [upload=476491]: Upload finished. err: <nil> Duration: 30.003975462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....] duration: 5.766748248s trace: https://jaeger.dev.storj.io/trace/68f38418308958ea FAIL (2025-12-28 06:49:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded ======LOGS====== 17m22.768855999s) [upload=476437]: Done starting upload Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments... Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment... Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment. Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments... Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces. Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments... Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil> Duration: 5.766773465s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X..XX.X...XX.....X................X.X.....X.X.........X.X...] duration: 5.001016023s trace: https://jaeger.dev.storj.io/trace/2230181a8dada927 FAIL (2025-12-28 06:49:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001041851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X..XX.X...XX.....X................X.X.....X.X.........X.X..] duration: 5.000546278s trace: https://jaeger.dev.storj.io/trace/2230181a8dada927 FAIL (2025-12-28 06:49:12 UTC) orbiter warning: context deadline exceeded Duration: 5.00057347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX....] duration: 5.938142682s trace: https://jaeger.dev.storj.io/trace/68f38418308958ea FAIL (2025-12-28 06:49:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded ======LOGS====== 17m22.768855999s) [upload=476437]: Done starting upload Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments... Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment... Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment. Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments... Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces. Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments... Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil> Duration: 5.938183329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X..XX.X...XX.....X................X.X.....X.X.........X.X.] duration: 3.441231063s trace: https://jaeger.dev.storj.io/trace/2230181a8dada927 FAIL (2025-12-28 06:48:24 UTC) orbiter warning: context deadline exceeded Duration: 3.441262053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX...] duration: 9.849922848s trace: https://jaeger.dev.storj.io/trace/68f38418308958ea FAIL (2025-12-28 06:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded ======LOGS====== 17m22.768855999s) [upload=476437]: Done starting upload Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments... Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment... Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment. Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece... Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments... Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil> Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces. Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments... Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil> Duration: 9.849951464s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)