Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX....X...XX...X.XX.X...X.....X..X........X...............] duration: 3.859673518s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:57:51 UTC) orbiter warning: context deadline exceeded Duration: 3.859702294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.732181029s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:57:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.732197359s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.643759761s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 15:57:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 6.643790329s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XX....X...XX...X.XX.X...X.....X..X........X..............] duration: 4.328104385s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:57:25 UTC) orbiter warning: context deadline exceeded Duration: 4.328129984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 7.374514716s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:56:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 7.374541298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 10.525134427s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 15:56:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 10.525163794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.791342928s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 15:56:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 5.791374269s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X...XX....X...XX...X.XX.X...X.....X..X........X.............] duration: 3.847131838s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:56:19 UTC) orbiter warning: context deadline exceeded Duration: 3.847233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.835097597s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:56:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.835123065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 16.825232791s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 15:55:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 16.825259893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X...XX....X...XX...X.XX.X...X.....X..X........X............] duration: 3.434239697s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:55:45 UTC) orbiter warning: context deadline exceeded Duration: 3.434264395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.096187694s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:55:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 6.096211831s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.228626489s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 15:55:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 8.228655944s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 11.426460075s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 15:55:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 11.426490774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.559804269s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 15:55:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 5.559837022s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.831044497s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:54:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.831076229s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X.X...XX....X...XX...X.XX.X...X.....X..X........X...........] duration: 3.609306852s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:54:40 UTC) orbiter warning: context deadline exceeded Duration: 3.609328763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 18.028119302s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 15:54:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 18.028160942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.381274066s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 15:54:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 6.381311448s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 25.054603869s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 15:54:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 25.05462533s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X.X...XX....X...XX...X.XX.X...X.....X..X........X..........] duration: 3.618215184s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:53:47 UTC) orbiter warning: context deadline exceeded Duration: 3.618239601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003171047s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 15:53:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 1m0.003194943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.802901346s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.802930341s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 18.616266881s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 15:53:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 18.616283723s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003624656s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 15:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 1m0.003647188s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.897446339s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:52:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.897474063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.989103583s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:52:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.989130533s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X.X...XX....X...XX...X.XX.X...X.....X..X........X.........] duration: 3.577312317s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:52:32 UTC) orbiter warning: context deadline exceeded Duration: 3.577589293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.809161127s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:52:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.809187147s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [...X.X...XX....X...XX...X.XX.X...X.....X..X........X........] duration: 3.181165493s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:52:12 UTC) orbiter warning: context deadline exceeded Duration: 3.18120072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [....X.X...XX....X...XX...X.XX.X...X.....X..X........X.......] duration: 3.638021615s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:51:27 UTC) orbiter warning: context deadline exceeded Duration: 3.638047136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.862818429s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:51:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.862846823s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.934550652s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:50:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.934584747s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X....X.X...XX....X...XX...X.XX.X...X.....X..X........X......] duration: 5.000948279s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:50:35 UTC) orbiter warning: context deadline exceeded Duration: 5.000977686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.895353054s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:50:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.895385707s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XX....X.X...XX....X...XX...X.XX.X...X.....X..X........X.....] duration: 3.505339736s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:49:38 UTC) orbiter warning: context deadline exceeded Duration: 3.505364052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.831372264s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:49:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.831464128s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.XX....X.X...XX....X...XX...X.XX.X...X.....X..X........X....] duration: 3.580932088s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:49:08 UTC) orbiter warning: context deadline exceeded Duration: 3.58095477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..XX....X.X...XX....X...XX...X.XX.X...X.....X..X........X...] duration: 3.558674804s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:48:42 UTC) orbiter warning: context deadline exceeded Duration: 3.558705874s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X..XX....X.X...XX....X...XX...X.XX.X...X.....X..X........X..] duration: 3.717496921s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:48:19 UTC) orbiter warning: context deadline exceeded Duration: 3.717519704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.199250953s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 6.199280629s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.X..XX....X.X...XX....X...XX...X.XX.X...X.....X..X........X.] duration: 3.565569663s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:47:54 UTC) orbiter warning: context deadline exceeded Duration: 3.565664774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.817818412s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:47:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.817847016s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X.X..XX....X.X...XX....X...XX...X.XX.X...X.....X..X........X] duration: 5.001600429s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 15:46:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001623913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.044559119s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:46:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 6.04459148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.X.X..XX....X.X...XX....X...XX...X.XX.X...X.....X..X........] duration: 3.679015976s trace: https://jaeger.dev.storj.io/trace/4be7ad0ba1c9f74d FAIL (2025-12-18 15:46:01 UTC) orbiter warning: context deadline exceeded Duration: 3.679047336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.822778588s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:45:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.822807091s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..X.X..XX....X.X...XX....X...XX...X.XX.X...X.....X..X.......] duration: 3.95379482s trace: https://jaeger.dev.storj.io/trace/4be7ad0ba1c9f74d FAIL (2025-12-18 15:45:17 UTC) orbiter warning: context deadline exceeded Duration: 3.953829607s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X.X..XX....X.X...XX....X...XX...X.XX.X...X.....X..X......] duration: 4.534480985s trace: https://jaeger.dev.storj.io/trace/4be7ad0ba1c9f74d FAIL (2025-12-18 15:44:58 UTC) orbiter warning: context deadline exceeded Duration: 4.534507586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.72688584s trace: https://jaeger.dev.storj.io/trace/1e6d8dbdb931ee38 FAIL (2025-12-18 15:44:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ) context deadline exceeded ======LOGS====== Dec 18 15:21:20.678038088 (311h52m20.549121133s) [upload=273478]: Starting upload Dec 18 15:21:20.678132587 (311h52m20.549215652s) [upload=273478]: Done starting upload Dec 18 15:21:20.678197040 (311h52m20.549280085s) [upload=273478]: Uploading segments... Dec 18 15:21:20.678377363 (311h52m20.549460408s) [upload=273478]: Got next segment. Inline: false Dec 18 15:21:20.678425085 (311h52m20.54950813s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 15:21:20.678431276 (311h52m20.549514331s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 15:21:20.930973077 (311h52m20.802056142s) [seg_pos={0 0}, upload=273478]: Begin upload segment... Dec 18 15:21:20.931042449 (311h52m20.802125494s) [seg_pos={0 0}, upload=273478]: Done begin upload segment. Dec 18 15:21:20.931145746 (311h52m20.802228791s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931167697 (311h52m20.802250742s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.931811143 (311h52m20.802894188s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932224610 (311h52m20.803307655s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.932623559 (311h52m20.803706604s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933042417 (311h52m20.804125462s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.933448100 (311h52m20.804531145s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934090623 (311h52m20.805173668s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.934632244 (311h52m20.805715289s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.936661067 (311h52m20.807744112s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.938865805 (311h52m20.80994885s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.941177999 (311h52m20.812261044s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.943096552 (311h52m20.814179597s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.944884536 (311h52m20.815967581s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.946736654 (311h52m20.817819699s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:20.948549415 (311h52m20.81963246s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Uploading piece... Dec 18 15:21:22.632579810 (311h52m22.503662845s) [upload=273478]: Next returned nil segment Dec 18 15:21:22.632600308 (311h52m22.503683354s) [upload=273478]: Waiting for error group managing segments... Dec 18 15:21:22.922505676 (311h52m22.793588711s) [noise=true , address=145.79.227.37:4567 , piece_id=M2XXWFUD4NL26OHBPCHIXF5IMTTGLVDZBDYRFJUVWGUF63WYRU3A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931058733 (311h52m22.802141778s) [noise=true , address=145.79.227.47:4265 , piece_id=QMBF4SFWT7H2KIBS4A5VHJYKKL6WHSORFX4CBNUUDHNP4OQYZS3Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.931939099 (311h52m22.803022144s) [noise=true , address=145.79.227.42:4122 , piece_id=6ISHRB3XGUTVAJGDURQFB76FAQWPFT3NIMOAOFLLM37SN5H7OETA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.934376330 (311h52m22.805459375s) [noise=true , address=145.79.227.46:4602 , piece_id=7OZEUDDNHNJQEN5FAZN4M7SYOKTSQJ3LMUAAW2JDQBNA7T7FYCAA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.941486559 (311h52m22.812569594s) [noise=true , address=145.79.227.40:4050 , piece_id=VWMBLPICZ4WT7HKDOQEWNUVL5E7SN7SFK7WTKJXM33OI2DEI5VUA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:22.956328852 (311h52m22.827411907s) [noise=true , address=145.79.227.18:4006 , piece_id=K3SK3UKBR2XLZ5UCTLJI5P2YQTK6HV3GTUWL3RCKYJU7X33UGQ6A, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.031740039 (311h52m22.902823074s) [noise=true , address=145.79.227.49:4337 , piece_id=LKBK2A3VHO6PL2V5AZS6JOZT23MLOBHVCG7VVG6MUY4634FZGARQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.034574386 (311h52m22.905657421s) [noise=true , address=145.79.227.9:4366 , piece_id=2XLZH75J5LWVNVI3INSFHWELVN2KQOCW7KMSRJPW5NJZAAY24O6Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.038955118 (311h52m22.910038163s) [noise=true , address=145.79.227.33:4871 , piece_id=4SKSDUQKMB2LAIETWJJQIJQTE5GYB6S2DDTHUIKTBJ6V2M46SSGQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.167104940 (311h52m23.038187975s) [noise=true , address=45.91.116.4:4940 , piece_id=7F5UCAHKGCQVL6NYATG3F5ZRDOHXTSFEAE24S7KOXFM6G2VVDDEQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.177051079 (311h52m23.048134124s) [noise=true , address=45.91.116.14:4192 , piece_id=RW34SJXMRBV4RFTVALQO4X2MAGI5MLHSVSNUH3YY7LTKOKFUC4DQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.201153325 (311h52m23.07223636s) [noise=true , address=45.91.116.12:4226 , piece_id=2GNP3LNS24CMDSLPEBOA57YMVYGNLWUJIZKZHNTHRA72ZP2OX2CQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264322501 (311h52m23.135405546s) [noise=true , address=77.90.31.12:4590 , piece_id=T3WETEEEHOA7U7IKKBWO66K5CXV3ECTXXLV5PTDH4TQBJTR6BAWQ, seg_pos={0 0}, upload=273478]: Done uploading piece. err: <nil> Dec 18 15:21:23.264343091 (311h52m23.135426136s) [seg_pos={0 0}, upload=273478]: Segment reached optimal threshold of 13 pieces. Dec 18 15:21:23.264587526 (311h52m23.135670561s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264596713 (311h52m23.135679749s) [noise=true , address=45.91.116.5:4298 , piece_id=EA7YK7P4OTIA5OZKGPWPEGCXQYH56BXSTF5FHIBAMLMMFKIVRECA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264784311 (311h52m23.135867356s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264801614 (311h52m23.135884649s) [noise=true , address=45.91.116.9:4264 , piece_id=WZPZVYE2HBCVBS3N42MQR4I23ILMQ7CXPCR4RNLO2XEPM7C4FZ7Q, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264853784 (311h52m23.135936819s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 18 15:21:23.264860907 (311h52m23.135943952s) [noise=true , address=145.79.227.44:4195 , piece_id=GTGUMGPOZLW5H6CCY7JLVRFHLJIGTTNS2GQJD25KBWPW7PWUYVGA, seg_pos={0 0}, upload=273478]: Long tail cancellation due to optimal threshold Dec 18 15:21:23.264878911 (311h52m23.135961946s) [seg_pos={0 0}, upload=273478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 15:21:23.264956298 (311h52m23.136039343s) [upload=273478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:21:23.264961088 (311h52m23.136044133s) [upload=273478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:21:23.589760999 (311h52m23.460844055s) [upload=273478]: Done uploading segments... Dec 18 15:21:23.589782651 (311h52m23.460865686s) [upload=273478]: Upload finished. err: <nil> Duration: 5.726914044s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)